is there mathematical model to describe the relationship between running time and input data size for hadoop? -


in hadoop cluster, there mathematical model describe curve transmission time , datainputsize of mapper?

for example, if original data size n m mappers, , total transmission time mappers reducers t. wanna double data size 2n in mappers, there approximation estimation transmission time t'(i think t' must less 2t), idea use log curve describe curve, not sure correct.

i assume input coming hdfs(?) assume input data has been placed on hdfs, we're not talking time transmit input data local file store hdfs. assume input size n total size of of input files combined. assume m number of map tasks (based on number of input splits input files broken into). if we're talking transmission between map tasks , reduce tasks need know size of output map operations. in general, size of output unrelated size of input n.

even if knew how total data needs transmitted between map tasks , reduce tasks, asking transmission time not meaningful, because transmission can happen @ same time map , reduce tasks executing, , series of individual transmissions between individual map tasks , reduce tasks each happen @ different points in time. goal of written hadoop application hide transmission time overlapping computation , communication.


Comments

Post a Comment

Popular posts from this blog

SPSS keyboard combination alters encoding -

my question not have programming practical issue on spss. accident keyword combination cannot remember because did not intend press changed encoding in spss , text in data in different language english shown small boxes. nevertheless, when kind of analysis performed different language text shown in output. does know reason? tried change encoding did not produce result. first, @ edit > options , see whether setting unicode or locale. might fix it. if have text in multiple languages, want unicode. otherwise, may have specific locale set wrong. can change set locale, need know locale want use.
Read more

Add new record to the table by click on the button in Microsoft Access -

in ms access form want implement separate button, adds new record table. in order added button , attached button event: private sub btnaddrec_click() refresh codecontextobject on error resume next docmd.gotorecord , , acnewrec if err.number <> 0 btnaddrec.enabled = false end if end end sub everything ok when open window , click btnaddrec button, problem when first of perform navigation through existed records , after click on button. got runtime error: 2105: «you can't go specified record. may @ end of recordset» . how solve issue, need have ability add new record on click on specific button, no matter, have walked or not walked through records before. thanks. i created simple form field call description (and autonumber) , created button code follows click event. filled number of records , navigated through them, clicked addnewrec button. form navigated new record without issues. able click addnewrec button directly after ...
Read more

CSS3 Transition to highlight new elements created in JQuery -

i want use css3 color transition apply highlight-fading color effect (yellow transparent) new elements appended markup using jquery. css #content div { background-color:transparent; -moz-transition:background-color 2s; -webkit-transition:background-color 2s; -o-transition:background-color 2s; transition:background-color 2s; } #content div.new { background-color:yellow; } html <div id="content"></div> <a id="add-element" href="#">add new element</a> js $('#add-element').click(function() { var newelement = $('<div class="new">new element</div>'); $('#content').append(newelement); newelement.removeclass('new'); }); when click link, new element created. class "new" (background color yellow) , it's appended html markup. should able remove "new" class trigger background color transition transparent (or whate...
Read more