Move incoming and outgoing flow computation to DuckDB

[abelsiqueira]

Related to #895

[github-actions]

Benchmark Results

	f27f85e...	9b3c863...	f27f85e.../9b3c8639383b0b...
energy_problem/create_model	43.8 ± 2 s	43.7 ± 0.71 s	1
energy_problem/input_and_constructor	11 ± 0.51 s	0.0187 ± 0.00012 h	0.164
time_to_load	3.72 ± 0.032 s	3.71 ± 0.03 s	1

	f27f85e...	9b3c863...	f27f85e.../9b3c8639383b0b...
energy_problem/create_model	0.647 G allocs: 21.9 GB	0.603 G allocs: 20.9 GB	1.05
energy_problem/input_and_constructor	31.2 M allocs: 1.54 GB	31.2 M allocs: 1.55 GB	0.993
time_to_load	0.157 k allocs: 11.1 kB	0.157 k allocs: 11.1 kB	1

Benchmark Plots

A plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR.
Go to "Actions"->"Benchmark a pull request"->[the most recent run]->"Artifacts" (at the bottom).

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.17%. Comparing base (f27f85e) to head (9b3c863).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #919   +/-   ##
=======================================
  Coverage   99.17%   99.17%           
=======================================
  Files          29       29           
  Lines        1205     1215   +10     
=======================================
+ Hits         1195     1205   +10     
  Misses         10       10           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[abelsiqueira]

@datejada @suvayu This introduces the big issue of the past: computing the indices of the incoming (and outgoing) flows is slow:

julia> using TimerOutputs;
julia> TulipaEnergyModel.to = TimerOutput();
julia> begin
         using TulipaEnergyModel, TulipaIO, DuckDB, DataFrames
         con = DBInterface.connect(DuckDB.DB)
         schemas = TulipaEnergyModel.schema_per_table_name
         read_csv_folder(con, "benchmark/EU/"; schemas)
         ep = EnergyProblem(con)
         create_model!(ep)
       end
Precompiling TulipaEnergyModel...
  2 dependencies successfully precompiled in 8 seconds. 122 already precompiled.
EnergyProblem:
  - Time creating internal structures (in seconds): 8.04610275
  - Time computing constraints partitions (in seconds): 4.152481208
  - Time creating dataframes (in seconds): 41.245806959
  - Time creating variables indices (in seconds): 2.7459e-5
  - Model created!
    - Time for  creating the model (in seconds): 42.290705958
    - Number of variables: 1556419
    - Number of constraints for variable bounds: 1248359
    - Number of structural constraints: 2303880
  - Model not solved!


julia> TulipaEnergyModel.to
────────────────────────────────────────────────────────────────────────────────────────────────
                                                       Time                    Allocations
                                              ───────────────────────   ────────────────────────
              Tot / % measured:                     120s /  68.9%           26.8GiB /  82.9%

Section                               ncalls     time    %tot     avg     alloc    %tot      avg
────────────────────────────────────────────────────────────────────────────────────────────────
create_model                               1    41.3s   49.9%   41.3s   22.1GiB   99.2%  22.1GiB
  add_expression_terms_to_df               1    12.1s   14.6%   12.1s   8.69GiB   39.1%  8.69GiB
  add_capacity_constraints!                1    10.7s   12.9%   10.7s   5.34GiB   24.0%  5.34GiB
  add_storage_constraints!                 1    3.46s    4.2%   3.46s   1.90GiB    8.5%  1.90GiB
  objective                                1    3.02s    3.7%   3.02s   1.05GiB    4.7%  1.05GiB
  add_flow_variables!                      1    2.90s    3.5%   2.90s   1.70GiB    7.7%  1.70GiB
  add_transport_constraints!               1    2.87s    3.5%   2.87s   1.55GiB    7.0%  1.55GiB
  add_storage_variables!                   1    996ms    1.2%   996ms    414MiB    1.8%   414MiB
  multi-year investment expressions        1    663ms    0.8%   663ms    109MiB    0.5%   109MiB
  add_consumer_constraints!                1    656ms    0.8%   656ms    317MiB    1.4%   317MiB
  add_hub_constraints!                     1    570ms    0.7%   570ms    447MiB    2.0%   447MiB
  unpacking dataframes                     1    408ms    0.5%   408ms   75.8MiB    0.3%  75.8MiB
  add_investment_variables!                1    317ms    0.4%   317ms   10.7MiB    0.0%  10.7MiB
  add_expressions_for_storage              1    113ms    0.1%   113ms   23.0MiB    0.1%  23.0MiB
  add_conversion_constraints!              1   94.8ms    0.1%  94.8ms   25.4MiB    0.1%  25.4MiB
  add_unit_commitment_variables!           1   85.6ms    0.1%  85.6ms   2.49MiB    0.0%  2.49MiB
  add_investment_constraints!              1   39.7ms    0.0%  39.7ms   1.95MiB    0.0%  1.95MiB
  add_energy_constraints!                  1   40.6μs    0.0%  40.6μs     64.0B    0.0%    64.0B
Create constraints indices                 1    39.8s   48.1%   39.8s   34.5KiB    0.0%  34.5KiB
  Create t_incoming_nonzero                1    29.1s   35.2%   29.1s   4.70KiB    0.0%  4.70KiB
  Create t_outgoing_nonzero                1    10.5s   12.6%   10.5s   4.70KiB    0.0%  4.70KiB
  Create highest_in_out_incoming           1   65.4ms    0.1%  65.4ms   4.70KiB    0.0%  4.70KiB
  Create highest_in_out_outgoing           1   35.3ms    0.0%  35.3ms   4.70KiB    0.0%  4.70KiB
unpacking and creating sets                1    1.01s    1.2%   1.01s    188MiB    0.8%   188MiB
Create partition tables                    1    641ms    0.8%   641ms   5.38MiB    0.0%  5.38MiB
────────────────────────────────────────────────────────────────────────────────────────────────

Now the goal is to improve (if possible) the creation of table t_incoming_nonzero.