Skip to content

Sparse + Parallel - #4

Open
rtbs-dev wants to merge 12 commits into
mainfrom
sparsify
Open

Sparse + Parallel#4
rtbs-dev wants to merge 12 commits into
mainfrom
sparsify

Conversation

@rtbs-dev

Copy link
Copy Markdown
Owner

Updates the core associations.py to use pydata/sparse as a backend

  • add flat2sq and sq2flat as sparse squareform re-writes
  • dispatch norm_diag to preserve sparsity in sparse grammian norms (e.g. cosine-sim)
  • parallelize core forest_pursuit loop with joblib

All together, we can do precision matrix/edge support reconstruction on sparse binary input with very little memory change from the pre-existing data.

10k variables and 50k observations can be made into a dependency graph estimate in under 10 sec on a laptop.

@rtbs-dev

Copy link
Copy Markdown
Owner Author

Unit tests still failing for strange shapes, but otherwise working for forest_pursuit_cts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant