Fossil SCM
Extended pass InitCsets and underlying code with more log output geared towards memory introspection, and added markers for special locations. Extended my notes with general observations from the first test runs over my example CVS repositories.
Commit
27ed4f7dc3a0c032f255bc0d6a3734c00f68a65d
Parent
7b71f64766ac808…
3 files changed
+47
-1
+23
+15
-2
+47
-1
| --- cvs2fossil.txt | ||
| +++ cvs2fossil.txt | ||
| @@ -7,10 +7,56 @@ | ||
| 7 | 7 | tcllib, or a single subproject of tcllib, like tklib, but not |
| 8 | 8 | multiple sub-projects in one go. |
| 9 | 9 | |
| 10 | 10 | * We have to look into the pass 'InitCsets' and hunt for the |
| 11 | 11 | cause of the large amount of memory it is gobbling up. |
| 12 | + | |
| 13 | + Results from the first look using the new memory tracking | |
| 14 | + subsystem: | |
| 15 | + | |
| 16 | + (1) The general architecture, workflow, is a bit wasteful. All | |
| 17 | + changesets are generated and kept in memory before getting | |
| 18 | + persisted. This means that allocated memory piles up over | |
| 19 | + time, with later changesets pushing the boundaries. This | |
| 20 | + is made worse that some of the preliminary changesets seem | |
| 21 | + to require a lot of temporary memory as part of getting | |
| 22 | + broken down into the actual ones. InititializeBreakState | |
| 23 | + seems to be the culprit here. Its memory usage is possibly | |
| 24 | + quadratic in the number of items in the changeset. | |
| 25 | + | |
| 26 | + (2) A number of small inefficiencies. Like 'state eval' always | |
| 27 | + pulling the whole result into memory before processing it | |
| 28 | + with 'foreach'. Here potentially large lists. | |
| 29 | + | |
| 30 | + (3) We maintain an in-memory map from tagged items to their | |
| 31 | + changesets. While this is needed later in the sorting | |
| 32 | + passes during the creation this is wasted space. And also | |
| 33 | + wasted time, to maintain it during the creation and | |
| 34 | + breaking. | |
| 35 | + | |
| 36 | + Changes: | |
| 37 | + | |
| 38 | + (a) Re-architect to create, break, and persist changesets one | |
| 39 | + by one, completely releasing all associated in-memory data | |
| 40 | + before going to the next. Should be low-hanging fruit with | |
| 41 | + high impact, as we have all the necessary operations | |
| 42 | + already, just not in that order, and that alone should | |
| 43 | + already keep the pile from forming, making the spikes of | |
| 44 | + (2) more manageable. | |
| 45 | + | |
| 46 | + (b) Look into the smaller problems described in (2), and | |
| 47 | + especially (3). These should still be low-hanging fruit, | |
| 48 | + although of lesser effect than (a). For (3) disable the | |
| 49 | + map and its maintenace during construction, and put it | |
| 50 | + into a separate command, to be used when loading the | |
| 51 | + created changesets at the end. | |
| 52 | + | |
| 53 | + (c) With larger effect, but more difficult to achieve, go into | |
| 54 | + command 'InitializeBreakState' and the preceding | |
| 55 | + 'internalsuccessors', and rearchitect it. Definitely not a | |
| 56 | + low-hanging fruit. Possibly also something we can skip if | |
| 57 | + doing (a) had a large enough effect. | |
| 12 | 58 | |
| 13 | 59 | * Look at the dependencies on external packages and consider |
| 14 | 60 | which of them can be moved into the importer, either as a |
| 15 | 61 | simple utility command, or wholesale. |
| 16 | 62 | |
| @@ -37,6 +83,6 @@ | ||
| 37 | 83 | |
| 38 | 84 | snit |
| 39 | 85 | In toto |
| 40 | 86 | |
| 41 | 87 | sqlite3 |
| 42 | - In tota | |
| 88 | + In toto | |
| 43 | 89 |
| --- cvs2fossil.txt | |
| +++ cvs2fossil.txt | |
| @@ -7,10 +7,56 @@ | |
| 7 | tcllib, or a single subproject of tcllib, like tklib, but not |
| 8 | multiple sub-projects in one go. |
| 9 | |
| 10 | * We have to look into the pass 'InitCsets' and hunt for the |
| 11 | cause of the large amount of memory it is gobbling up. |
| 12 | |
| 13 | * Look at the dependencies on external packages and consider |
| 14 | which of them can be moved into the importer, either as a |
| 15 | simple utility command, or wholesale. |
| 16 | |
| @@ -37,6 +83,6 @@ | |
| 37 | |
| 38 | snit |
| 39 | In toto |
| 40 | |
| 41 | sqlite3 |
| 42 | In tota |
| 43 |
| --- cvs2fossil.txt | |
| +++ cvs2fossil.txt | |
| @@ -7,10 +7,56 @@ | |
| 7 | tcllib, or a single subproject of tcllib, like tklib, but not |
| 8 | multiple sub-projects in one go. |
| 9 | |
| 10 | * We have to look into the pass 'InitCsets' and hunt for the |
| 11 | cause of the large amount of memory it is gobbling up. |
| 12 | |
| 13 | Results from the first look using the new memory tracking |
| 14 | subsystem: |
| 15 | |
| 16 | (1) The general architecture, workflow, is a bit wasteful. All |
| 17 | changesets are generated and kept in memory before getting |
| 18 | persisted. This means that allocated memory piles up over |
| 19 | time, with later changesets pushing the boundaries. This |
| 20 | is made worse that some of the preliminary changesets seem |
| 21 | to require a lot of temporary memory as part of getting |
| 22 | broken down into the actual ones. InititializeBreakState |
| 23 | seems to be the culprit here. Its memory usage is possibly |
| 24 | quadratic in the number of items in the changeset. |
| 25 | |
| 26 | (2) A number of small inefficiencies. Like 'state eval' always |
| 27 | pulling the whole result into memory before processing it |
| 28 | with 'foreach'. Here potentially large lists. |
| 29 | |
| 30 | (3) We maintain an in-memory map from tagged items to their |
| 31 | changesets. While this is needed later in the sorting |
| 32 | passes during the creation this is wasted space. And also |
| 33 | wasted time, to maintain it during the creation and |
| 34 | breaking. |
| 35 | |
| 36 | Changes: |
| 37 | |
| 38 | (a) Re-architect to create, break, and persist changesets one |
| 39 | by one, completely releasing all associated in-memory data |
| 40 | before going to the next. Should be low-hanging fruit with |
| 41 | high impact, as we have all the necessary operations |
| 42 | already, just not in that order, and that alone should |
| 43 | already keep the pile from forming, making the spikes of |
| 44 | (2) more manageable. |
| 45 | |
| 46 | (b) Look into the smaller problems described in (2), and |
| 47 | especially (3). These should still be low-hanging fruit, |
| 48 | although of lesser effect than (a). For (3) disable the |
| 49 | map and its maintenace during construction, and put it |
| 50 | into a separate command, to be used when loading the |
| 51 | created changesets at the end. |
| 52 | |
| 53 | (c) With larger effect, but more difficult to achieve, go into |
| 54 | command 'InitializeBreakState' and the preceding |
| 55 | 'internalsuccessors', and rearchitect it. Definitely not a |
| 56 | low-hanging fruit. Possibly also something we can skip if |
| 57 | doing (a) had a large enough effect. |
| 58 | |
| 59 | * Look at the dependencies on external packages and consider |
| 60 | which of them can be moved into the importer, either as a |
| 61 | simple utility command, or wholesale. |
| 62 | |
| @@ -37,6 +83,6 @@ | |
| 83 | |
| 84 | snit |
| 85 | In toto |
| 86 | |
| 87 | sqlite3 |
| 88 | In toto |
| 89 |
| --- tools/cvs2fossil/lib/c2f_pinitcsets.tcl | ||
| +++ tools/cvs2fossil/lib/c2f_pinitcsets.tcl | ||
| @@ -19,10 +19,11 @@ | ||
| 19 | 19 | |
| 20 | 20 | package require Tcl 8.4 ; # Required runtime. |
| 21 | 21 | package require snit ; # OO system. |
| 22 | 22 | package require vc::tools::misc ; # Text formatting. |
| 23 | 23 | package require vc::tools::log ; # User feedback. |
| 24 | +package require vc::tools::mem ; # Memory tracking. | |
| 24 | 25 | package require vc::fossil::import::cvs::repository ; # Repository management. |
| 25 | 26 | package require vc::fossil::import::cvs::state ; # State storage. |
| 26 | 27 | package require vc::fossil::import::cvs::integrity ; # State integrity checks. |
| 27 | 28 | package require vc::fossil::import::cvs::project::rev ; # Project level changesets |
| 28 | 29 | |
| @@ -179,21 +180,29 @@ | ||
| 179 | 180 | # early, extending them with all their revisions. This |
| 180 | 181 | # however would mean lots of (slow) method invokations |
| 181 | 182 | # on the csets. Doing it like this, late creation, means |
| 182 | 183 | # less such calls. None, but the creation itself. |
| 183 | 184 | |
| 185 | + log write 14 initcsets meta_begin | |
| 186 | + mem::mark | |
| 184 | 187 | foreach {mid rid pid} [state run { |
| 185 | 188 | SELECT M.mid, R.rid, M.pid |
| 186 | 189 | FROM revision R, meta M -- R ==> M, using PK index of M. |
| 187 | 190 | WHERE R.mid = M.mid |
| 188 | 191 | ORDER BY M.mid, R.date |
| 189 | 192 | }] { |
| 193 | + log write 14 initcsets meta_next | |
| 194 | + | |
| 190 | 195 | if {$lastmeta != $mid} { |
| 191 | 196 | if {[llength $revisions]} { |
| 192 | 197 | incr n |
| 193 | 198 | set p [repository projectof $lastproject] |
| 199 | + log write 14 initcsets meta_cset_begin | |
| 200 | + mem::mark | |
| 194 | 201 | project::rev %AUTO% $p rev $lastmeta $revisions |
| 202 | + log write 14 initcsets meta_cset_done | |
| 203 | + mem::mark | |
| 195 | 204 | set revisions {} |
| 196 | 205 | } |
| 197 | 206 | set lastmeta $mid |
| 198 | 207 | set lastproject $pid |
| 199 | 208 | } |
| @@ -201,19 +210,27 @@ | ||
| 201 | 210 | } |
| 202 | 211 | |
| 203 | 212 | if {[llength $revisions]} { |
| 204 | 213 | incr n |
| 205 | 214 | set p [repository projectof $lastproject] |
| 215 | + log write 14 initcsets meta_cset_begin | |
| 216 | + mem::mark | |
| 206 | 217 | project::rev %AUTO% $p rev $lastmeta $revisions |
| 218 | + log write 14 initcsets meta_cset_done | |
| 219 | + mem::mark | |
| 207 | 220 | } |
| 221 | + | |
| 222 | + log write 14 initcsets meta_done | |
| 223 | + mem::mark | |
| 208 | 224 | |
| 209 | 225 | log write 4 initcsets "Created [nsp $n {revision changeset}]" |
| 210 | 226 | return |
| 211 | 227 | } |
| 212 | 228 | |
| 213 | 229 | proc CreateSymbolChangesets {} { |
| 214 | 230 | log write 3 initcsets {Create changesets based on symbols} |
| 231 | + mem::mark | |
| 215 | 232 | |
| 216 | 233 | # Tags and branches induce changesets as well, containing the |
| 217 | 234 | # revisions they are attached to (tags), or spawned from |
| 218 | 235 | # (branches). |
| 219 | 236 | |
| @@ -279,10 +296,11 @@ | ||
| 279 | 296 | set p [repository projectof $lastproject] |
| 280 | 297 | project::rev %AUTO% $p sym::branch $lastsymbol $branches |
| 281 | 298 | } |
| 282 | 299 | |
| 283 | 300 | log write 4 initcsets "Created [nsp $n {symbol changeset}]" |
| 301 | + mem::mark | |
| 284 | 302 | return |
| 285 | 303 | } |
| 286 | 304 | |
| 287 | 305 | proc BreakInternalDependencies {} { |
| 288 | 306 | # This code operates on the revision changesets created by |
| @@ -292,19 +310,21 @@ | ||
| 292 | 310 | # by splitting the problematic changeset into multiple |
| 293 | 311 | # fragments. The results are changesets which have no internal |
| 294 | 312 | # dependencies, only external ones. |
| 295 | 313 | |
| 296 | 314 | log write 3 initcsets {Break internal dependencies} |
| 315 | + mem::mark | |
| 297 | 316 | set old [llength [project::rev all]] |
| 298 | 317 | |
| 299 | 318 | foreach cset [project::rev all] { |
| 300 | 319 | $cset breakinternaldependencies |
| 301 | 320 | } |
| 302 | 321 | |
| 303 | 322 | set n [expr {[llength [project::rev all]] - $old}] |
| 304 | 323 | log write 4 initcsets "Created [nsp $n {additional revision changeset}]" |
| 305 | 324 | log write 4 initcsets Ok. |
| 325 | + mem::mark | |
| 306 | 326 | return |
| 307 | 327 | } |
| 308 | 328 | |
| 309 | 329 | proc PersistTheChangesets {} { |
| 310 | 330 | log write 3 initcsets "Saving [nsp [llength [project::rev all]] {initial changeset}] to the persistent state" |
| @@ -334,10 +354,13 @@ | ||
| 334 | 354 | namespace import ::vc::fossil::import::cvs::state |
| 335 | 355 | namespace import ::vc::fossil::import::cvs::integrity |
| 336 | 356 | namespace eval project { |
| 337 | 357 | namespace import ::vc::fossil::import::cvs::project::rev |
| 338 | 358 | } |
| 359 | + namespace eval mem { | |
| 360 | + namespace import ::vc::tools::mem::mark | |
| 361 | + } | |
| 339 | 362 | namespace import ::vc::tools::misc::* |
| 340 | 363 | namespace import ::vc::tools::log |
| 341 | 364 | log register initcsets |
| 342 | 365 | } |
| 343 | 366 | } |
| 344 | 367 |
| --- tools/cvs2fossil/lib/c2f_pinitcsets.tcl | |
| +++ tools/cvs2fossil/lib/c2f_pinitcsets.tcl | |
| @@ -19,10 +19,11 @@ | |
| 19 | |
| 20 | package require Tcl 8.4 ; # Required runtime. |
| 21 | package require snit ; # OO system. |
| 22 | package require vc::tools::misc ; # Text formatting. |
| 23 | package require vc::tools::log ; # User feedback. |
| 24 | package require vc::fossil::import::cvs::repository ; # Repository management. |
| 25 | package require vc::fossil::import::cvs::state ; # State storage. |
| 26 | package require vc::fossil::import::cvs::integrity ; # State integrity checks. |
| 27 | package require vc::fossil::import::cvs::project::rev ; # Project level changesets |
| 28 | |
| @@ -179,21 +180,29 @@ | |
| 179 | # early, extending them with all their revisions. This |
| 180 | # however would mean lots of (slow) method invokations |
| 181 | # on the csets. Doing it like this, late creation, means |
| 182 | # less such calls. None, but the creation itself. |
| 183 | |
| 184 | foreach {mid rid pid} [state run { |
| 185 | SELECT M.mid, R.rid, M.pid |
| 186 | FROM revision R, meta M -- R ==> M, using PK index of M. |
| 187 | WHERE R.mid = M.mid |
| 188 | ORDER BY M.mid, R.date |
| 189 | }] { |
| 190 | if {$lastmeta != $mid} { |
| 191 | if {[llength $revisions]} { |
| 192 | incr n |
| 193 | set p [repository projectof $lastproject] |
| 194 | project::rev %AUTO% $p rev $lastmeta $revisions |
| 195 | set revisions {} |
| 196 | } |
| 197 | set lastmeta $mid |
| 198 | set lastproject $pid |
| 199 | } |
| @@ -201,19 +210,27 @@ | |
| 201 | } |
| 202 | |
| 203 | if {[llength $revisions]} { |
| 204 | incr n |
| 205 | set p [repository projectof $lastproject] |
| 206 | project::rev %AUTO% $p rev $lastmeta $revisions |
| 207 | } |
| 208 | |
| 209 | log write 4 initcsets "Created [nsp $n {revision changeset}]" |
| 210 | return |
| 211 | } |
| 212 | |
| 213 | proc CreateSymbolChangesets {} { |
| 214 | log write 3 initcsets {Create changesets based on symbols} |
| 215 | |
| 216 | # Tags and branches induce changesets as well, containing the |
| 217 | # revisions they are attached to (tags), or spawned from |
| 218 | # (branches). |
| 219 | |
| @@ -279,10 +296,11 @@ | |
| 279 | set p [repository projectof $lastproject] |
| 280 | project::rev %AUTO% $p sym::branch $lastsymbol $branches |
| 281 | } |
| 282 | |
| 283 | log write 4 initcsets "Created [nsp $n {symbol changeset}]" |
| 284 | return |
| 285 | } |
| 286 | |
| 287 | proc BreakInternalDependencies {} { |
| 288 | # This code operates on the revision changesets created by |
| @@ -292,19 +310,21 @@ | |
| 292 | # by splitting the problematic changeset into multiple |
| 293 | # fragments. The results are changesets which have no internal |
| 294 | # dependencies, only external ones. |
| 295 | |
| 296 | log write 3 initcsets {Break internal dependencies} |
| 297 | set old [llength [project::rev all]] |
| 298 | |
| 299 | foreach cset [project::rev all] { |
| 300 | $cset breakinternaldependencies |
| 301 | } |
| 302 | |
| 303 | set n [expr {[llength [project::rev all]] - $old}] |
| 304 | log write 4 initcsets "Created [nsp $n {additional revision changeset}]" |
| 305 | log write 4 initcsets Ok. |
| 306 | return |
| 307 | } |
| 308 | |
| 309 | proc PersistTheChangesets {} { |
| 310 | log write 3 initcsets "Saving [nsp [llength [project::rev all]] {initial changeset}] to the persistent state" |
| @@ -334,10 +354,13 @@ | |
| 334 | namespace import ::vc::fossil::import::cvs::state |
| 335 | namespace import ::vc::fossil::import::cvs::integrity |
| 336 | namespace eval project { |
| 337 | namespace import ::vc::fossil::import::cvs::project::rev |
| 338 | } |
| 339 | namespace import ::vc::tools::misc::* |
| 340 | namespace import ::vc::tools::log |
| 341 | log register initcsets |
| 342 | } |
| 343 | } |
| 344 |
| --- tools/cvs2fossil/lib/c2f_pinitcsets.tcl | |
| +++ tools/cvs2fossil/lib/c2f_pinitcsets.tcl | |
| @@ -19,10 +19,11 @@ | |
| 19 | |
| 20 | package require Tcl 8.4 ; # Required runtime. |
| 21 | package require snit ; # OO system. |
| 22 | package require vc::tools::misc ; # Text formatting. |
| 23 | package require vc::tools::log ; # User feedback. |
| 24 | package require vc::tools::mem ; # Memory tracking. |
| 25 | package require vc::fossil::import::cvs::repository ; # Repository management. |
| 26 | package require vc::fossil::import::cvs::state ; # State storage. |
| 27 | package require vc::fossil::import::cvs::integrity ; # State integrity checks. |
| 28 | package require vc::fossil::import::cvs::project::rev ; # Project level changesets |
| 29 | |
| @@ -179,21 +180,29 @@ | |
| 180 | # early, extending them with all their revisions. This |
| 181 | # however would mean lots of (slow) method invokations |
| 182 | # on the csets. Doing it like this, late creation, means |
| 183 | # less such calls. None, but the creation itself. |
| 184 | |
| 185 | log write 14 initcsets meta_begin |
| 186 | mem::mark |
| 187 | foreach {mid rid pid} [state run { |
| 188 | SELECT M.mid, R.rid, M.pid |
| 189 | FROM revision R, meta M -- R ==> M, using PK index of M. |
| 190 | WHERE R.mid = M.mid |
| 191 | ORDER BY M.mid, R.date |
| 192 | }] { |
| 193 | log write 14 initcsets meta_next |
| 194 | |
| 195 | if {$lastmeta != $mid} { |
| 196 | if {[llength $revisions]} { |
| 197 | incr n |
| 198 | set p [repository projectof $lastproject] |
| 199 | log write 14 initcsets meta_cset_begin |
| 200 | mem::mark |
| 201 | project::rev %AUTO% $p rev $lastmeta $revisions |
| 202 | log write 14 initcsets meta_cset_done |
| 203 | mem::mark |
| 204 | set revisions {} |
| 205 | } |
| 206 | set lastmeta $mid |
| 207 | set lastproject $pid |
| 208 | } |
| @@ -201,19 +210,27 @@ | |
| 210 | } |
| 211 | |
| 212 | if {[llength $revisions]} { |
| 213 | incr n |
| 214 | set p [repository projectof $lastproject] |
| 215 | log write 14 initcsets meta_cset_begin |
| 216 | mem::mark |
| 217 | project::rev %AUTO% $p rev $lastmeta $revisions |
| 218 | log write 14 initcsets meta_cset_done |
| 219 | mem::mark |
| 220 | } |
| 221 | |
| 222 | log write 14 initcsets meta_done |
| 223 | mem::mark |
| 224 | |
| 225 | log write 4 initcsets "Created [nsp $n {revision changeset}]" |
| 226 | return |
| 227 | } |
| 228 | |
| 229 | proc CreateSymbolChangesets {} { |
| 230 | log write 3 initcsets {Create changesets based on symbols} |
| 231 | mem::mark |
| 232 | |
| 233 | # Tags and branches induce changesets as well, containing the |
| 234 | # revisions they are attached to (tags), or spawned from |
| 235 | # (branches). |
| 236 | |
| @@ -279,10 +296,11 @@ | |
| 296 | set p [repository projectof $lastproject] |
| 297 | project::rev %AUTO% $p sym::branch $lastsymbol $branches |
| 298 | } |
| 299 | |
| 300 | log write 4 initcsets "Created [nsp $n {symbol changeset}]" |
| 301 | mem::mark |
| 302 | return |
| 303 | } |
| 304 | |
| 305 | proc BreakInternalDependencies {} { |
| 306 | # This code operates on the revision changesets created by |
| @@ -292,19 +310,21 @@ | |
| 310 | # by splitting the problematic changeset into multiple |
| 311 | # fragments. The results are changesets which have no internal |
| 312 | # dependencies, only external ones. |
| 313 | |
| 314 | log write 3 initcsets {Break internal dependencies} |
| 315 | mem::mark |
| 316 | set old [llength [project::rev all]] |
| 317 | |
| 318 | foreach cset [project::rev all] { |
| 319 | $cset breakinternaldependencies |
| 320 | } |
| 321 | |
| 322 | set n [expr {[llength [project::rev all]] - $old}] |
| 323 | log write 4 initcsets "Created [nsp $n {additional revision changeset}]" |
| 324 | log write 4 initcsets Ok. |
| 325 | mem::mark |
| 326 | return |
| 327 | } |
| 328 | |
| 329 | proc PersistTheChangesets {} { |
| 330 | log write 3 initcsets "Saving [nsp [llength [project::rev all]] {initial changeset}] to the persistent state" |
| @@ -334,10 +354,13 @@ | |
| 354 | namespace import ::vc::fossil::import::cvs::state |
| 355 | namespace import ::vc::fossil::import::cvs::integrity |
| 356 | namespace eval project { |
| 357 | namespace import ::vc::fossil::import::cvs::project::rev |
| 358 | } |
| 359 | namespace eval mem { |
| 360 | namespace import ::vc::tools::mem::mark |
| 361 | } |
| 362 | namespace import ::vc::tools::misc::* |
| 363 | namespace import ::vc::tools::log |
| 364 | log register initcsets |
| 365 | } |
| 366 | } |
| 367 |
+15
-2
| --- tools/cvs2fossil/lib/c2f_prev.tcl | ||
| +++ tools/cvs2fossil/lib/c2f_prev.tcl | ||
| @@ -133,11 +133,12 @@ | ||
| 133 | 133 | $mytypeobj successors tmp $myitems |
| 134 | 134 | return [array get tmp] |
| 135 | 135 | } |
| 136 | 136 | |
| 137 | 137 | method breakinternaldependencies {} { |
| 138 | - | |
| 138 | + log write 14 csets {[$self str] BID} | |
| 139 | + vc::tools::mem::mark | |
| 139 | 140 | ## |
| 140 | 141 | ## NOTE: This method, maybe in conjunction with its caller |
| 141 | 142 | ## seems to be a memory hog, especially for large |
| 142 | 143 | ## changesets, with 'large' meaning to have a 'long list |
| 143 | 144 | ## of items, several thousand'. Investigate where the |
| @@ -166,13 +167,16 @@ | ||
| 166 | 167 | # Array of dependencies (parent -> child). This is pulled from |
| 167 | 168 | # the state, and limited to successors within the changeset. |
| 168 | 169 | |
| 169 | 170 | array set dependencies {} |
| 170 | 171 | $mytypeobj internalsuccessors dependencies $myitems |
| 171 | - if {![array size dependencies]} {return 0} ; # Nothing to break. | |
| 172 | + if {![array size dependencies]} { | |
| 173 | + return 0 | |
| 174 | + } ; # Nothing to break. | |
| 172 | 175 | |
| 173 | 176 | log write 5 csets ...[$self str]....................................................... |
| 177 | + vc::tools::mem::mark | |
| 174 | 178 | |
| 175 | 179 | # We have internal dependencies to break. We now iterate over |
| 176 | 180 | # all positions in the list (which is chronological, at least |
| 177 | 181 | # as far as the timestamps are correct and unique) and |
| 178 | 182 | # determine the best position for the break, by trying to |
| @@ -1059,10 +1063,12 @@ | ||
| 1059 | 1063 | |
| 1060 | 1064 | # var(dv) = dict (revision -> list (revision)) |
| 1061 | 1065 | typemethod internalsuccessors {dv revisions} { |
| 1062 | 1066 | upvar 1 $dv dependencies |
| 1063 | 1067 | set theset ('[join $revisions {','}]') |
| 1068 | + | |
| 1069 | + log write 14 cset internalsuccessors | |
| 1064 | 1070 | |
| 1065 | 1071 | # See 'successors' below for the main explanation of |
| 1066 | 1072 | # the various cases. This piece is special in that it |
| 1067 | 1073 | # restricts the successors we look for to the same set of |
| 1068 | 1074 | # revisions we start from. Sensible as we are looking for |
| @@ -1121,10 +1127,12 @@ | ||
| 1121 | 1127 | |
| 1122 | 1128 | # We allow revisions to be far apart in time in the same |
| 1123 | 1129 | # changeset, but in turn need the pseudo-dependencies to |
| 1124 | 1130 | # handle this. |
| 1125 | 1131 | |
| 1132 | + log write 14 cset pseudo-internalsuccessors | |
| 1133 | + | |
| 1126 | 1134 | array set fids {} |
| 1127 | 1135 | foreach {rid fid} [state run [subst -nocommands -nobackslashes { |
| 1128 | 1136 | SELECT R.rid, R.fid |
| 1129 | 1137 | FROM revision R |
| 1130 | 1138 | WHERE R.rid IN $theset |
| @@ -1141,10 +1149,12 @@ | ||
| 1141 | 1149 | set dep($a,$b) . |
| 1142 | 1150 | set dep($b,$a) . |
| 1143 | 1151 | } |
| 1144 | 1152 | } |
| 1145 | 1153 | } |
| 1154 | + | |
| 1155 | + log write 14 cset complete | |
| 1146 | 1156 | return |
| 1147 | 1157 | } |
| 1148 | 1158 | |
| 1149 | 1159 | # result = 4-list (itemtype itemid nextitemtype nextitemid ...) |
| 1150 | 1160 | typemethod loops {revisions} { |
| @@ -1611,22 +1621,25 @@ | ||
| 1611 | 1621 | |
| 1612 | 1622 | # Set up the helper singletons |
| 1613 | 1623 | namespace eval rev { |
| 1614 | 1624 | namespace import ::vc::fossil::import::cvs::state |
| 1615 | 1625 | namespace import ::vc::fossil::import::cvs::integrity |
| 1626 | + namespace import ::vc::tools::log | |
| 1616 | 1627 | } |
| 1617 | 1628 | namespace eval sym::tag { |
| 1618 | 1629 | namespace import ::vc::fossil::import::cvs::state |
| 1619 | 1630 | namespace import ::vc::fossil::import::cvs::integrity |
| 1631 | + namespace import ::vc::tools::log | |
| 1620 | 1632 | } |
| 1621 | 1633 | namespace eval sym::branch { |
| 1622 | 1634 | namespace import ::vc::fossil::import::cvs::state |
| 1623 | 1635 | namespace import ::vc::fossil::import::cvs::integrity |
| 1636 | + namespace import ::vc::tools::log | |
| 1624 | 1637 | } |
| 1625 | 1638 | } |
| 1626 | 1639 | } |
| 1627 | 1640 | |
| 1628 | 1641 | # # ## ### ##### ######## ############# ##################### |
| 1629 | 1642 | ## Ready |
| 1630 | 1643 | |
| 1631 | 1644 | package provide vc::fossil::import::cvs::project::rev 1.0 |
| 1632 | 1645 | return |
| 1633 | 1646 |
| --- tools/cvs2fossil/lib/c2f_prev.tcl | |
| +++ tools/cvs2fossil/lib/c2f_prev.tcl | |
| @@ -133,11 +133,12 @@ | |
| 133 | $mytypeobj successors tmp $myitems |
| 134 | return [array get tmp] |
| 135 | } |
| 136 | |
| 137 | method breakinternaldependencies {} { |
| 138 | |
| 139 | ## |
| 140 | ## NOTE: This method, maybe in conjunction with its caller |
| 141 | ## seems to be a memory hog, especially for large |
| 142 | ## changesets, with 'large' meaning to have a 'long list |
| 143 | ## of items, several thousand'. Investigate where the |
| @@ -166,13 +167,16 @@ | |
| 166 | # Array of dependencies (parent -> child). This is pulled from |
| 167 | # the state, and limited to successors within the changeset. |
| 168 | |
| 169 | array set dependencies {} |
| 170 | $mytypeobj internalsuccessors dependencies $myitems |
| 171 | if {![array size dependencies]} {return 0} ; # Nothing to break. |
| 172 | |
| 173 | log write 5 csets ...[$self str]....................................................... |
| 174 | |
| 175 | # We have internal dependencies to break. We now iterate over |
| 176 | # all positions in the list (which is chronological, at least |
| 177 | # as far as the timestamps are correct and unique) and |
| 178 | # determine the best position for the break, by trying to |
| @@ -1059,10 +1063,12 @@ | |
| 1059 | |
| 1060 | # var(dv) = dict (revision -> list (revision)) |
| 1061 | typemethod internalsuccessors {dv revisions} { |
| 1062 | upvar 1 $dv dependencies |
| 1063 | set theset ('[join $revisions {','}]') |
| 1064 | |
| 1065 | # See 'successors' below for the main explanation of |
| 1066 | # the various cases. This piece is special in that it |
| 1067 | # restricts the successors we look for to the same set of |
| 1068 | # revisions we start from. Sensible as we are looking for |
| @@ -1121,10 +1127,12 @@ | |
| 1121 | |
| 1122 | # We allow revisions to be far apart in time in the same |
| 1123 | # changeset, but in turn need the pseudo-dependencies to |
| 1124 | # handle this. |
| 1125 | |
| 1126 | array set fids {} |
| 1127 | foreach {rid fid} [state run [subst -nocommands -nobackslashes { |
| 1128 | SELECT R.rid, R.fid |
| 1129 | FROM revision R |
| 1130 | WHERE R.rid IN $theset |
| @@ -1141,10 +1149,12 @@ | |
| 1141 | set dep($a,$b) . |
| 1142 | set dep($b,$a) . |
| 1143 | } |
| 1144 | } |
| 1145 | } |
| 1146 | return |
| 1147 | } |
| 1148 | |
| 1149 | # result = 4-list (itemtype itemid nextitemtype nextitemid ...) |
| 1150 | typemethod loops {revisions} { |
| @@ -1611,22 +1621,25 @@ | |
| 1611 | |
| 1612 | # Set up the helper singletons |
| 1613 | namespace eval rev { |
| 1614 | namespace import ::vc::fossil::import::cvs::state |
| 1615 | namespace import ::vc::fossil::import::cvs::integrity |
| 1616 | } |
| 1617 | namespace eval sym::tag { |
| 1618 | namespace import ::vc::fossil::import::cvs::state |
| 1619 | namespace import ::vc::fossil::import::cvs::integrity |
| 1620 | } |
| 1621 | namespace eval sym::branch { |
| 1622 | namespace import ::vc::fossil::import::cvs::state |
| 1623 | namespace import ::vc::fossil::import::cvs::integrity |
| 1624 | } |
| 1625 | } |
| 1626 | } |
| 1627 | |
| 1628 | # # ## ### ##### ######## ############# ##################### |
| 1629 | ## Ready |
| 1630 | |
| 1631 | package provide vc::fossil::import::cvs::project::rev 1.0 |
| 1632 | return |
| 1633 |
| --- tools/cvs2fossil/lib/c2f_prev.tcl | |
| +++ tools/cvs2fossil/lib/c2f_prev.tcl | |
| @@ -133,11 +133,12 @@ | |
| 133 | $mytypeobj successors tmp $myitems |
| 134 | return [array get tmp] |
| 135 | } |
| 136 | |
| 137 | method breakinternaldependencies {} { |
| 138 | log write 14 csets {[$self str] BID} |
| 139 | vc::tools::mem::mark |
| 140 | ## |
| 141 | ## NOTE: This method, maybe in conjunction with its caller |
| 142 | ## seems to be a memory hog, especially for large |
| 143 | ## changesets, with 'large' meaning to have a 'long list |
| 144 | ## of items, several thousand'. Investigate where the |
| @@ -166,13 +167,16 @@ | |
| 167 | # Array of dependencies (parent -> child). This is pulled from |
| 168 | # the state, and limited to successors within the changeset. |
| 169 | |
| 170 | array set dependencies {} |
| 171 | $mytypeobj internalsuccessors dependencies $myitems |
| 172 | if {![array size dependencies]} { |
| 173 | return 0 |
| 174 | } ; # Nothing to break. |
| 175 | |
| 176 | log write 5 csets ...[$self str]....................................................... |
| 177 | vc::tools::mem::mark |
| 178 | |
| 179 | # We have internal dependencies to break. We now iterate over |
| 180 | # all positions in the list (which is chronological, at least |
| 181 | # as far as the timestamps are correct and unique) and |
| 182 | # determine the best position for the break, by trying to |
| @@ -1059,10 +1063,12 @@ | |
| 1063 | |
| 1064 | # var(dv) = dict (revision -> list (revision)) |
| 1065 | typemethod internalsuccessors {dv revisions} { |
| 1066 | upvar 1 $dv dependencies |
| 1067 | set theset ('[join $revisions {','}]') |
| 1068 | |
| 1069 | log write 14 cset internalsuccessors |
| 1070 | |
| 1071 | # See 'successors' below for the main explanation of |
| 1072 | # the various cases. This piece is special in that it |
| 1073 | # restricts the successors we look for to the same set of |
| 1074 | # revisions we start from. Sensible as we are looking for |
| @@ -1121,10 +1127,12 @@ | |
| 1127 | |
| 1128 | # We allow revisions to be far apart in time in the same |
| 1129 | # changeset, but in turn need the pseudo-dependencies to |
| 1130 | # handle this. |
| 1131 | |
| 1132 | log write 14 cset pseudo-internalsuccessors |
| 1133 | |
| 1134 | array set fids {} |
| 1135 | foreach {rid fid} [state run [subst -nocommands -nobackslashes { |
| 1136 | SELECT R.rid, R.fid |
| 1137 | FROM revision R |
| 1138 | WHERE R.rid IN $theset |
| @@ -1141,10 +1149,12 @@ | |
| 1149 | set dep($a,$b) . |
| 1150 | set dep($b,$a) . |
| 1151 | } |
| 1152 | } |
| 1153 | } |
| 1154 | |
| 1155 | log write 14 cset complete |
| 1156 | return |
| 1157 | } |
| 1158 | |
| 1159 | # result = 4-list (itemtype itemid nextitemtype nextitemid ...) |
| 1160 | typemethod loops {revisions} { |
| @@ -1611,22 +1621,25 @@ | |
| 1621 | |
| 1622 | # Set up the helper singletons |
| 1623 | namespace eval rev { |
| 1624 | namespace import ::vc::fossil::import::cvs::state |
| 1625 | namespace import ::vc::fossil::import::cvs::integrity |
| 1626 | namespace import ::vc::tools::log |
| 1627 | } |
| 1628 | namespace eval sym::tag { |
| 1629 | namespace import ::vc::fossil::import::cvs::state |
| 1630 | namespace import ::vc::fossil::import::cvs::integrity |
| 1631 | namespace import ::vc::tools::log |
| 1632 | } |
| 1633 | namespace eval sym::branch { |
| 1634 | namespace import ::vc::fossil::import::cvs::state |
| 1635 | namespace import ::vc::fossil::import::cvs::integrity |
| 1636 | namespace import ::vc::tools::log |
| 1637 | } |
| 1638 | } |
| 1639 | } |
| 1640 | |
| 1641 | # # ## ### ##### ######## ############# ##################### |
| 1642 | ## Ready |
| 1643 | |
| 1644 | package provide vc::fossil::import::cvs::project::rev 1.0 |
| 1645 | return |
| 1646 |