-
Notifications
You must be signed in to change notification settings - Fork 7
/
machine.info.irene.html
7497 lines (7254 loc) · 589 KB
/
machine.info.irene.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>irene.info v0.0</title>
<script type="text/javascript" src="_static/js/modernizr.min.js"></script>
<script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/language_data.js"></script>
<script type="text/javascript" src="_static/js/theme.js"></script>
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="#" class="icon icon-home"> irene
<img src="_static/tgcc-logo.png" class="logo" alt="Logo"/>
</a>
<div class="version">
2021-12-01.1214
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<!-- Local TOC -->
<div class="local-toc"><p class="caption"><span class="caption-text">Content</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Introduction">Introduction</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#the-computing-center">The computing center</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#access-to-computing-resources">Access to computing resources</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Supercomputer_architecture">Supercomputer architecture</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#configuration-of-machine">Configuration of Irene</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#interconnect">Interconnect</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#lustre">Lustre</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#quick-start-for-amd-rome">Quick start for AMD Rome</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#quick-start-for-arm-a64fx">Quick start for ARM A64FX</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/User_account">User account</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#user-account-creation">User account creation</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#passwords">Passwords</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#account-locking-and-account-deletion">Account locking and account deletion</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Interactive_access">Interactive access</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#system-access">System access</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#login-nodes-usage">Login nodes usage</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#remote-desktop-system-service-nicedcv">Remote Desktop System service (NiceDCV)</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Data_spaces">Data spaces</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#available-file-systems">Available file systems</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#quota">Quota</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#data-protection">Data protection</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#personal-spaces">Personal Spaces</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#shared-spaces">Shared Spaces</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#parallel-file-system-usage-monitoring">Parallel file system usage monitoring</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Data_transfer">Data transfers</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#file-transfer">File transfer</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#synchronize-your-ccrt-account-with-your-prace-account">Synchronize your CCRT account with your PRACE account</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#ccfr-infrastructure">CCFR infrastructure</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#prace-infrastructure">PRACE infrastructure</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Environment_management">Environment management</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#what-is-module">What is module</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#module-actions">Module actions</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#initialization-and-scope">Initialization and scope</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#major-modulefiles">Major modulefiles</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#extend-your-environment-with-modulefiles">Extend your environment with modulefiles</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Software">Softwares</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#generalities-on-software">Generalities on software</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#specific-software">Specific software</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#product-life-cycle">Product Life cycle</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Job_submission">Job submission</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#scheduling-policy">Scheduling policy</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#choosing-the-file-systems">Choosing the file systems</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#submission-scripts">Submission scripts</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#job-monitoring-and-control">Job monitoring and control</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#special-jobs">Special jobs</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Project_accounting">Project accounting</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#computing-hours-consumption-control-process">Computing hours consumption control process</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#ccc-myproject">ccc_myproject</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#ccc-compuse">ccc_compuse</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Compiling">Compilation</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#language-standard">Language standard</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#available-compilers">Available compilers</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#available-numerical-libraries">Available numerical libraries</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#compiling-for-skylake">Compiling for Skylake</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#compiling-for-rome-milan">Compiling for Rome/Milan</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Parallel_programming">Parallel programming</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#mpi">MPI</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#openmp">OpenMP</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#using-gpus">Using GPUs</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Runtime_tuning">Runtime tuning</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#memory-allocation-tuning">Memory allocation tuning</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Process_distribution_affinity_binding">Process distribution, affinity and binding</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#hardware-topology">Hardware topology</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#definitions">Definitions</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#process-distribution">Process distribution</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#process-and-thread-affinity">Process and thread affinity</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#hyper-threading-usage">Hyper-Threading usage</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#turbo">Turbo</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/IO">Parallel IO</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#mpi-io">MPI-IO</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#recommended-data-usage-on-parallel-file-system">Recommended data usage on parallel file system</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#parallel-compression-and-decompression-with-pigz">Parallel compression and decompression with pigz</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Debugging">Debugging</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#summary">Summary</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#compiler-flags">Compiler flags</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#available-debuggers">Available debuggers</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#other-tools">Other tools</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Profiling">Profiling</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#summary">Summary</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#ipm">IPM</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#darshan">Darshan</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#arm-forge-map">Arm-forge MAP</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#gprof">Gprof</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#igprof">igprof</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#papi">PAPI</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#scalasca">Scalasca</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#vampir">Vampir</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#extra-p">Extra-P</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#hpctoolkit">HPCToolkit</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#intel-vtune">Intel Vtune</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#valgrind">Valgrind</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#intel-advisor">Intel Advisor</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#tau-tuning-and-analysis-utilities">TAU (Tuning and Analysis Utilities)</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#gprof2dot">Gprof2dot</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#perf">Perf</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#memonit">Memonit</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Post-processing">Post-processing</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#gnuplot">Gnuplot</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#xmgrace">Xmgrace</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#tecplot">Tecplot</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#ensight">Ensight</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#visit">visit</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#paraview">paraview</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Virtualization">Virtualization</a><ul>
<li class="toctree-l2"><a class="reference internal" href="irene.html#pcocc">PCOCC</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#pre-requisites-for-virtual-machines">Pre-requisites for virtual machines</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#launching-a-cluster-of-vms">Launching a cluster of VMs</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#bridge-templates">Bridge templates</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#customizing-the-vm">Customizing the VM</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#bridge-plugin">Bridge plugin</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#checkpoint-restart-with-bridge-plugin">Checkpoint / restart with Bridge plugin</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#launching-a-cluster-of-containers">Launching a cluster of containers</a></li>
<li class="toctree-l2"><a class="reference internal" href="irene.html#whole-set-of-commands">Whole set of commands</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="irene.html#document-toc/fulldoc/Glossary">Glossary of CCC commands</a></li>
</ul>
</div>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="#">irene</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="#">Docs</a> »</li>
<li>irene.info v0.0</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="welcome-to-irene-documentation">
<h1>Welcome to Irene documentation<a class="headerlink" href="#welcome-to-irene-documentation" title="Permalink to this headline">¶</a></h1>
<p>This user guide is provided by the TGCC computing center (<a class="reference external" href="https://www-tgcc.ccc.cea.fr/">https://www-tgcc.ccc.cea.fr/</a>).</p>
<div class="line-block">
<div class="line">This documentation can be found in different formats:</div>
<div class="line"><br /></div>
<div class="line">HTML:</div>
<div class="line"><a class="reference external" href="../../fr/html/irene.html">Documentation en français HTML</a></div>
<div class="line"><a class="reference external" href="../../fr/singlehtml/irene.html">Documentation en français HTML (Page unique)</a></div>
<div class="line"><a class="reference external" href="../../en/html/irene.html">English documentation HTML</a></div>
<div class="line"><a class="reference external" href="../../en/singlehtml/irene.html">English documentation HTML (One page)</a></div>
<div class="line"><br /></div>
<div class="line">PDF:</div>
<div class="line"><a class="reference external" href="../../fr/latex/irene.pdf">Documentation en français PDF</a></div>
<div class="line"><a class="reference external" href="../../en/latex/irene.pdf">English documentation PDF</a></div>
<div class="line"><br /></div>
</div>
<p>This documentation is available on clusters with the following commands:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ machine.info
$ man irene
</pre></div>
</div>
<p>Feel free to contact us at <a class="reference external" href="mailto:hotline.tgcc%40cea.fr">hotline<span>.</span>tgcc<span>@</span>cea<span>.</span>fr</a> for any suggestion about this documentation.</p>
<div class="toctree-wrapper compound">
<span id="document-toc/fulldoc/Introduction"></span><div class="section" id="introduction">
<span id="id1"></span><h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
<div class="section" id="the-computing-center">
<h3>The computing center<a class="headerlink" href="#the-computing-center" title="Permalink to this headline">¶</a></h3>
<p>The <abbr title="Très Grand Centre de Calcul du CEA">TGCC</abbr> is a high performance computing infrastructure aiming at hosting state-of-the-art supercomputers in France.</p>
<p>It is designed to:</p>
<ul class="simple">
<li>Accommodate future high end computing systems.</li>
<li>Provide a communication and exhibition space for scientific events (conferences, seminars, training sessions, …).</li>
<li>Propose a flexible and modular facility for future evolution of HPC systems.</li>
</ul>
<p>The TGCC hosts two clusters:</p>
<ul class="simple">
<li>Joliot-Curie, a 21 PFlops supercomputer that is part of <abbr title="Grand Equipement National de Calcul Intensif">GENCI</abbr>. It represents the french contribution to the European <abbr title="Partnership for Advanced Computing in Europe">PRACE</abbr> infrastructure.</li>
<li>Topaze, a 4 PFlops supercomputer that is part of the <abbr title="Centre de Calcul Recherche et Technologie, used by the CEA and its industrial partners">CCRT</abbr>.</li>
</ul>
<p>The computing center architecture is data-centric. The computing nodes are connected to the private Lustre storage system for very fast I/O. And Global Lustre filesystems are shared between the different supercomputers. A hierarchical data storage system manages petabytes of data and is used for long term storage and archiving of results.</p>
<p>The computing center operates round the clock, except for scheduled maintenance periods when the teams update the hardware, firmware and software. The regular maintenance periods also allow a check of the general behavior and performance of the various supercomputers. Every year, a compulsory electrical maintenance requires a full shut down of the supercomputers for a longer period (3 days).</p>
<p>On-site support for the contractors, expert administration teams and an on-call duty system optimize the availability of the computing service. Guaranteeing access security and data confidentiality is a major preoccupation. A CEA IT security experts unit controls a security supervision system which monitors, detects and analyzes security alerts, enabling the security managers to react extremely rapidly.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p>The Hotline is the single point of contact for any question or support request:</p>
<ul class="simple">
<li>mail : <a class="reference external" href="mailto:hotline.tgcc%40cea.fr">hotline<span>.</span>tgcc<span>@</span>cea<span>.</span>fr</a></li>
<li>tel : +33 1 77 57 42 42</li>
</ul>
<p class="last">Depending on its type, the demand will be transmitted to the appropriate team.</p>
</div>
<p>A users committee (COMUT) takes place every trimester to exchange between users and the TGCC staff. The representatives of each community can provide their feedback on the general usage of the computing center. Any big change or update is announced during the COMUT.</p>
<p>Training sessions are organized by TGCC staff on a regular basis.</p>
</div>
<div class="section" id="access-to-computing-resources">
<h3>Access to computing resources<a class="headerlink" href="#access-to-computing-resources" title="Permalink to this headline">¶</a></h3>
<p>Computing hours on the computing center are granted in a community-dependent fashion:</p>
<ul>
<li><p class="first">CCRT Partners automatically get a share of the computing resources.</p>
</li>
<li><p class="first">French research teams can ask for resources through GENCI thanks to DARI calls: <a class="reference external" href="http://www.edari.fr">http://www.edari.fr</a></p>
</li>
<li><p class="first">Scientists and researchers from academia and industry can ask for resources through PRACE. PRACE accesses are of 2 kinds:</p>
<ul class="simple">
<li>Preparatory Access is intended for short-term access to resources, for code-enabling and porting, required to prepare proposals for Project Access and to demonstrate the scalability of codes. Applications for Preparatory Access are accepted at any time, with a cut-off date every 3 months.</li>
<li>Project Access is intended for individual researchers and research groups including multinational research groups. It can be used for 1-year production runs, as well as for 2-year or 3-year (Multi-Year Access) production runs.</li>
</ul>
<blockquote>
<div><p>Project Access is subject to the PRACE Peer Review Process, which includes technical and scientific review. Technical experts and leading scientists evaluate the proposals submitted in response to the bi-annual calls. Applications for Preparatory Access undergo technical review only.
For more information on how to apply for access to PRACE resources, go to <a class="reference external" href="http://www.prace-ri.eu/how-to-apply">http://www.prace-ri.eu/how-to-apply</a></p>
</div></blockquote>
</li>
</ul>
<p>Ongoing PRACE projects are monitored as much as possible. Therefore, we regularly ask for your feedback. Feel free to tell us about any issue you are facing.</p>
<p>Project or partner accounts are granted an amount of computing hours or a computing share. They must use the awarded hours on a regular basis. To ensure that: over-consumption lowers priority so jobs might need more time to access resources, and reaching an under-consumption limit may result in hours being removed from a project.</p>
</div>
</div>
<span id="document-toc/fulldoc/Supercomputer_architecture"></span><div class="section" id="supercomputer-architecture">
<span id="id1"></span><h2>Supercomputer architecture<a class="headerlink" href="#supercomputer-architecture" title="Permalink to this headline">¶</a></h2>
<div class="section" id="configuration-of-machine">
<h3>Configuration of Irene<a class="headerlink" href="#configuration-of-machine" title="Permalink to this headline">¶</a></h3>
<p>The compute nodes are gathered in partitions according to their hardware characteristics (CPU architecture, amount of RAM, presence of GPU, etc). A partition is a set of identical nodes that can be targeted to host one or several jobs. Choosing the right partition for a job depends on code prerequisites in term of hardware resources. For example, executing a code designed to be GPU accelerated requires a partition with GPU nodes.</p>
<p>The Irene supercomputer offers three different kind of nodes: regular compute nodes, KNL nodes, large memory nodes and GPU nodes.</p>
<ul>
<li><dl class="first docutils">
<dt>Skylake nodes for regular computation</dt><dd><ul class="simple">
<li>Partition name: skylake</li>
<li>CPU : 2x24-cores Intel <a class="reference external" href="mailto:Skylake%402.7GHz">Skylake<span>@</span>2<span>.</span>7GHz</a> (AVX512)</li>
<li>Cores/Node: 48</li>
<li>Nodes: 1 656</li>
<li>Total cores: 79 488</li>
<li>RAM/Node: 180GB</li>
<li>RAM/Core: 3.75GB</li>
</ul>
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt>KNL nodes for regular computation</dt><dd><ul class="simple">
<li>Partition name: knl</li>
<li>CPUs: 1x68-cores Intel <a class="reference external" href="mailto:KNL%401.4GHz">KNL<span>@</span>1<span>.</span>4GHz</a></li>
<li>Cores/Node: 64 (With 4 additional cores reserved for the operating system. They are referenced by the scheduler but not taken into account for hours consumption.)</li>
<li>Nodes: 828</li>
<li>Total cores : 52 992</li>
<li>RAM/Node: 96GB</li>
<li>RAM/Core: 1.4GB</li>
<li>Cluster mode is set to quadrant</li>
<li>MCDRAM (Multi-Channel Dynamic Random Access Memory) is set as a last-level cache (cache mode)</li>
</ul>
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt>AMD Rome nodes for regular computation</dt><dd><ul class="simple">
<li>Partition name : Rome</li>
<li>CPUs: 2x64 AMD <a class="reference external" href="mailto:Rome%402.6Ghz">Rome<span>@</span>2<span>.</span>6Ghz</a> (AVX2)</li>
<li>Core/Node: 128</li>
<li>Nodes: 2292</li>
<li>Total core: 293 376</li>
<li>RAM/Node: 256GB</li>
<li>RAM/core : 2GB</li>
</ul>
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt>Hybrid nodes for GPU computing and graphical usage</dt><dd><ul class="simple">
<li>Partition name: hybrid</li>
<li>CPUs: 2x24-cores Intel <a class="reference external" href="mailto:Skylake%402.7GHz">Skylake<span>@</span>2<span>.</span>7GHz</a> (AVX512)</li>
<li>GPUs: 1x Nvidia Pascal P100</li>
<li>Cores/Node: 48</li>
<li>Nodes: 20</li>
<li>Total cores: 960 (+ 20 GPU)</li>
<li>RAM/Node: 192GB</li>
<li>RAM/Core: 4GB</li>
<li>I/O: 1 HDD 250 GB + 1 SSD 800 GB/NVMe</li>
</ul>
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt>Fat nodes with a lot of shared memory for computation lasting a reasonable amount of time and using no more than one node</dt><dd><ul class="simple">
<li>Partition name: xlarge</li>
<li>CPUs: 4x28-cores Intel <a class="reference external" href="mailto:Skylake%402.1GHz">Skylake<span>@</span>2<span>.</span>1GHz</a></li>
<li>GPUs: 1x Nvidia Pascal P100</li>
<li>Cores/Node: 112</li>
<li>Nodes: 5</li>
<li>Total cores: 560</li>
<li>RAM/Node: 3TB</li>
<li>RAM/Core: 27GB</li>
<li>IO: 2 HDD de 1 TB + 1 SSD 1600 GB/NVMe</li>
</ul>
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt>V100 nodes for GPU computing and AI</dt><dd><ul class="simple">
<li>Partition name: V100</li>
<li>CPUs: 2x20-cores Intel <a class="reference external" href="mailto:Cascadelake%402.1GHz">Cascadelake<span>@</span>2<span>.</span>1GHz</a> (AVX512)</li>
<li>GPUs: 4x Nvidia Tesla V100</li>
<li>Cores/Node: 40</li>
<li>Nodes: 32</li>
<li>Total cores: 1280 (+ 128 GPU)</li>
<li>RAM/Node: 180 GB</li>
<li>RAM/Core: 4.5 GB</li>
</ul>
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt>ARM A64FX for regular computation</dt><dd><ul class="simple">
<li>Partition name : A64FX</li>
<li>CPUs : 1x48 A64FX Armv8.2-A SVE @1.8Ghz</li>
<li>Core/Node : 48</li>
<li>Nodes : 80</li>
<li>Total core : 3840</li>
<li>RAM/Node : 32GB</li>
<li>RAM/core : 666MB</li>
</ul>
</dd>
</dl>
</li>
</ul>
<p><strong class="command">ccc_mpinfo</strong> displays the available partitions/queues that can be used on a job.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ccc_mpinfo
--------------CPUS------------ -------------NODES------------
PARTITION STATUS TOTAL DOWN USED FREE TOTAL DOWN USED FREE MpC CpN SpN CpS TpC
--------- ------ ------ ------ ------ ------ ------ ------ ------ ------ ----- --- --- --- ---
milan up 107776 0 104373 3403 842 0 821 21 1867 128 2 64 1
a100 up 6144 128 3840 2176 48 1 30 17 3867 128 2 64 1
xlarge up 256 0 0 256 2 0 0 2 31250 128 2 64 1
</pre></div>
</div>
<ul class="simple">
<li><strong>MpC</strong> : amount of memory per core</li>
<li><strong>CpN</strong> : number of cores per node</li>
<li><strong>SpN</strong> : number of sockets per node</li>
<li><strong>Cps</strong> : number of cores per socket</li>
<li><strong>TpC</strong> : number of threads per core (for hyperthreading)</li>
</ul>
</div>
<div class="section" id="interconnect">
<h3>Interconnect<a class="headerlink" href="#interconnect" title="Permalink to this headline">¶</a></h3>
<p>The compute nodes are connected through a EDR InfiniBand network in a pruned FAT tree topology. This high throughput (100GB/s) and low latency network is used for I/O and communications among nodes of the supercomputer.</p>
</div>
<div class="section" id="lustre">
<h3>Lustre<a class="headerlink" href="#lustre" title="Permalink to this headline">¶</a></h3>
<p>Lustre is a type of parallel distributed file system, commonly used for large-scale cluster computing. It actually relies on a set of multiple I/O servers and the Lustre software presents them as a single unified filesystem.</p>
<p>The major Lustre components are the <abbr title="MetaData Server">MDS</abbr> and <abbr title="Object Storage Servers">OSSs</abbr>. The MDS stores metadata such as file names, directories, access permissions, and file layout. It is not actually involved in any I/O operations. The actual data is stored on the OSSs. Note that one single file can be stored on several OSSs which is one of the benefits of Lustre when working with large files.</p>
<div class="figure align-default" id="id5">
<img alt="Lustre.png" src="_images/Lustre.png" />
<p class="caption"><span class="caption-text">Lustre.png</span><a class="headerlink" href="#id5" title="Permalink to this image">¶</a></p>
</div>
<p>More information on how Lustre works and best practices are described in <a class="reference internal" href="irene.html#recommended-data-usage-on-parallel-file-system"><span class="std std-ref">Lustre best practice</span></a>.</p>
</div>
<div class="section" id="quick-start-for-amd-rome">
<h3>Quick start for AMD Rome<a class="headerlink" href="#quick-start-for-amd-rome" title="Permalink to this headline">¶</a></h3>
<div class="section" id="compilation">
<h4>Compilation<a class="headerlink" href="#compilation" title="Permalink to this headline">¶</a></h4>
<p>By default intel compiler (icc/ifort) adds a test when building on intel processors. In order to avoid an error when building for execution on a Rome architecture, one should use the <code class="xref std std-option docutils literal notranslate"><span class="pre">-mavx2</span></code> flags for compilation. If the user wants to compile an optimized version that works on any processor of Irene (ie : skl, knl and rome), he will need to use the following flags <code class="xref std std-option docutils literal notranslate"><span class="pre">-mavx2</span> <span class="pre">-ax</span> <span class="pre">CORE-AVX512,MIC-AVX512</span></code>.</p>
</div>
<div class="section" id="blis-numerical-library">
<h4>BLIS numerical library<a class="headerlink" href="#blis-numerical-library" title="Permalink to this headline">¶</a></h4>
<p>BLIS is a portable open-source software framework for instantiating high-performance BLAS-like dense linear algebra libraries. The framework was designed to isolate essential kernels of computation that, when optimized, immediately enable optimized implementations of most of its commonly used and computationally intensive operations. Select kernels have been optimized for the AMD EPYC processor family by AMD and others.</p>
<p>To compile your code with this library, you need to load the blis module and to add the environment variable <span class="target" id="index-0"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">CCC_LDFLAGS</span></code>:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ module load blis
</pre></div>
</div>
<ul class="simple">
<li>if you use the fortran language, use the following compilation commands:</li>
</ul>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ ifort prog.f90 <span class="si">${</span><span class="nv">CCC_LDFLAGS</span><span class="si">}</span>
$ mpifort prog.f90 <span class="si">${</span><span class="nv">CCC_LDFLAGS</span><span class="si">}</span>
</pre></div>
</div>
<ul class="simple">
<li>if you use the c language, use the following compilation commands:</li>
</ul>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ icc prog.c <span class="si">${</span><span class="nv">CCC_LDFLAGS</span><span class="si">}</span>
$ mpicc prog.c <span class="si">${</span><span class="nv">CCC_LDFLAGS</span><span class="si">}</span>
</pre></div>
</div>
<p>BLIS also includes a BLAS compatibility layer which gives application developers access to BLIS implementations via traditional BLAS API calls, that can be used in FORTRAN as well as in C code. BLIS also provides a CBLAS API, which is a C-style interface for BLAS, that can be called from C code.</p>
</div>
<div class="section" id="processor-performance">
<h4>Processor performance<a class="headerlink" href="#processor-performance" title="Permalink to this headline">¶</a></h4>
<p>A Rome core has a lower max floating point performance than a skylake when taking vectorization into account (67 Gflop/s per core on skylake vs 41.6 Gflop/s per core on Rome).
For the non vectorial part, the performance per core is analogous on skylake and rome.</p>
</div>
<div class="section" id="memory-architecture">
<h4>Memory architecture<a class="headerlink" href="#memory-architecture" title="Permalink to this headline">¶</a></h4>
<p>The memory bandwith of a Rome node is around 320 GB/s max which imply 2.5GB/s per core ( for comparaison pupose a skylake has a 4GB/s bandwith)</p>
<p>The Rome processor has three levels of NUMA effects :</p>
<ul class="simple">
<li>by ccx : it consists of a group of 4 cores which directly share the L3 cache (and as such have a better access to a data already in cache)</li>
<li>by memory controller : the 8 memory channels of the socket are split in 4 dual channel controllers, each dedicated to a group of 4 ccx ( 16 core in total) which will have a better access to the memory attached to its controller</li>
<li>by socket : the 64 cores of a socket have a faster access to the memory attached to it than the one on the other socket.</li>
</ul>
<p>The consequences for hybrid MPI/OpenMP code is that a probable good configuration is one of the following :</p>
<ul class="simple">
<li>4 OpenMP by process if your code shared a lot of access between threads in L3 cache</li>
<li>16 OpenMP by process if your code is bound by memory latency</li>
<li>64 or 128 OpenMP by process if your code already use a first touch allocation</li>
</ul>
</div>
<div class="section" id="network">
<h4>Network<a class="headerlink" href="#network" title="Permalink to this headline">¶</a></h4>
<p>The network is a 100Gb/s shared between 128 cores. The bandwith per core is around 2.6 times lower than skylake which means that communication heavy full MPI codes may suffer a slowdown when compared to skylake.</p>
</div>
</div>
<div class="section" id="quick-start-for-arm-a64fx">
<h3>Quick start for ARM A64FX<a class="headerlink" href="#quick-start-for-arm-a64fx" title="Permalink to this headline">¶</a></h3>
<div class="section" id="id2">
<h4>Memory architecture<a class="headerlink" href="#id2" title="Permalink to this headline">¶</a></h4>
<p>The memory bandwith of an A64FX node is around 1024Gb/s max which imply ~21Gb/s per core (instead of 4GB/s per core for a skylake partition node).</p>
<p>Its processor has 4 memory controlers each attributed to a group of cores. Each group, called Core Memory Group (CMG), contains 12 contiguous cores that share a single L2 cache (8Mb). Within a CMG, access to data shared between threads is faster than between threads of different CMGs.</p>
<p>Hence one of the most efficient MPI / OpenMP distribution is usually 4 MPI processes with 12 OpenMP threads per node.</p>
</div>
<div class="section" id="id3">
<h4>Network<a class="headerlink" href="#id3" title="Permalink to this headline">¶</a></h4>
<p>Similarly to the Skylake and Rome partitions, the network bandwidth per node is 100Gb/s. Therefore, the A64FX partition has a network bandwidth per core of ~2Gb/s which is identical to the Skylake partition. In comparison with the Rome partition which has a network bandwidth per core of ~0.8Gb/s.</p>
</div>
<div class="section" id="ssl2-numerical-library">
<h4>SSL2 numerical library<a class="headerlink" href="#ssl2-numerical-library" title="Permalink to this headline">¶</a></h4>
<p>Fujitsu provides a dedicated numerical library, called Fujitsu Scientific Subroutine Library II (SSL2), optimized for A64FX. This library provides linear algebra functions like the Open Blas ones (see below for the associated compilation option).</p>
</div>
<div class="section" id="id4">
<h4>Compilation<a class="headerlink" href="#id4" title="Permalink to this headline">¶</a></h4>
<p>Fujitsu provide dedicated compilers for A64FX architecture:</p>
<ul class="simple">
<li><strong>fcc</strong> for C</li>
<li><strong>FCC</strong> for C ++</li>
<li><strong>frt</strong> for Fortran</li>
</ul>
<p>The compilation optimization option recommended by the manufacturor, in order to achieve performances, is: <code class="xref std std-option docutils literal notranslate"><span class="pre">-Kfast</span></code>.</p>
<p>Activation of the OpenMP parallelization is done with the option: <code class="xref std std-option docutils literal notranslate"><span class="pre">-Kopenmp</span></code>.
We can condense the two previous options into a single instruction: <code class="xref std std-option docutils literal notranslate"><span class="pre">-Kfast,openmp</span></code>.
Furthermore, the call to the SSL2 numerical library is made with the option: <code class="xref std std-option docutils literal notranslate"><span class="pre">-SSL2</span></code>.</p>
</div>
<div class="section" id="documentation">
<h4>Documentation<a class="headerlink" href="#documentation" title="Permalink to this headline">¶</a></h4>
<p>More information about Fujitsu compilers and error messages can be found in the Fujitsu documentation located at this path.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>/ccc/products/cdc_docs/fujitsu_compiler/
</pre></div>
</div>
</div>
</div>
</div>
<span id="document-toc/fulldoc/User_account"></span><div class="section" id="user-account">
<span id="id1"></span><h2>User account<a class="headerlink" href="#user-account" title="Permalink to this headline">¶</a></h2>
<p>The Irene supercomputer is accessible to scientific users and to CCRT partners.</p>
<p>Academic use is granted through French and European calls for projects:</p>
<ul class="simple">
<li>PRACE calls for proposals: <a class="reference external" href="http://www.prace-ri.eu/hpc-access">http://www.prace-ri.eu/hpc-access</a></li>
<li>DARI calls for proposals: <a class="reference external" href="http://www.edari.fr">http://www.edari.fr</a></li>
</ul>
<div class="section" id="user-account-creation">
<h3>User account creation<a class="headerlink" href="#user-account-creation" title="Permalink to this headline">¶</a></h3>
<p>Each user willing to access the supercomputer needs its own account. For obvious security reasons, you cannot have one account shared by several users.</p>
<p>To create an account, the project leader should fill in and send the form available on the TGCC website to the TGCC Hotline providing the following information:</p>
<ul class="simple">
<li>The users information (name, adress, phone number, email, etc)<ul>
<li>For internships, or CDD’s, the end of validity of the account is required</li>
</ul>
</li>
<li>The project information<ul>
<li>Project name and reference number for PRACE or GENCI projects</li>
<li>Partner name for CCRT accounts</li>
</ul>
</li>
<li>The official technical contact person</li>
<li>The safety officer</li>
<li>Security information</li>
</ul>
<p>The first 6 characters of your password
The connecting IP address or access mode (for CCRT: external CEA, VPN, IP network)</p>
<p>It is recommended to use the interface provided at <a class="reference external" href="https://www-dcc.extra.cea.fr/Tgcc/">https://www-dcc.extra.cea.fr/Tgcc/</a>. It allows you to edit the account creation form interactively. It should then be signed and transmitted to the indicated address.</p>
</div>
<div class="section" id="passwords">
<h3>Passwords<a class="headerlink" href="#passwords" title="Permalink to this headline">¶</a></h3>
<p>A default password is set when your account is created. You will be automatically asked to change it the first time you connect.</p>
<p>Once connected, the password change is done with the <strong class="command">passwd</strong> command:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ passwd
Changing password <span class="k">for</span> user <login>.
Current Password: <<span class="nb">type</span> here current password>
New password: <enter new password>
Retype new password: <retype same new password>
passwd: all authentication tokens updated successfully.
</pre></div>
</div>
<p>A password must comply with the following rules:</p>
<ul class="simple">
<li>be at least 12 characters long and use at least 3 distinct characters classes</li>
<li>be renewed after 12 months</li>
<li>previously used password cannot be reused</li>
<li>if you modify your password, you will need to wait 5 days before modifying it again</li>
</ul>
<p>You can check the expiration date of your password with the command <strong class="command">ccc_password_expiration</strong>:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ ccc_password_expiration
Password <span class="k">for</span> <login>: <<span class="nb">type</span> here current password>
Your password will expire in <span class="m">192</span> days on Fri Mar <span class="m">14</span> <span class="m">11</span>:04:16 <span class="m">2014</span>
</pre></div>
</div>
<p>If you loose your password, contact TGCC Hotline and ask for a password reset.</p>
</div>
<div class="section" id="account-locking-and-account-deletion">
<h3>Account locking and account deletion<a class="headerlink" href="#account-locking-and-account-deletion" title="Permalink to this headline">¶</a></h3>
<p>Accounts may be locked and deleted once it is not used anymore. Here are the rules regarding account locking:</p>
<ul class="simple">
<li>Accounts related to a project (PRACE or GENCI) are locked 30 days after the end of the project.</li>
<li>Any account that is inactive for 6 months will be locked. Activity refers to any connexion to the frontend or a job submission.</li>
<li>If an end date was provided when the account was created, the account will be locket on that date.</li>
<li>We may decide to lock an account temporarily if we consider its usage of the resources is inappropriate. For example, an account that uses too much memory on the login nodes or does not respect some limits on the file systems may be locked. In any case you will receive several warning emails before we actually lock the account.</li>
</ul>
<p>One year after being locked, accounts are <strong>deleted</strong> along with their data.</p>
<p>If your account is locked and you want to unlock it, please send an email to <a class="reference external" href="mailto:hotline.tgcc%40cea.fr">hotline<span>.</span>tgcc<span>@</span>cea<span>.</span>fr</a> with an half-password of 6 characters.</p>
</div>
</div>
<span id="document-toc/fulldoc/Interactive_access"></span><div class="section" id="interactive-access">
<span id="id1"></span><h2>Interactive access<a class="headerlink" href="#interactive-access" title="Permalink to this headline">¶</a></h2>
<div class="section" id="system-access">
<h3>System access<a class="headerlink" href="#system-access" title="Permalink to this headline">¶</a></h3>
<p>Any user or CCRT partner that has been allocated computing hours or a computing share can access the supercomputer through the Internet (or through its partner network) but there are some prerequisites:</p>
<ul class="simple">
<li>your account has to be created,</li>
<li>If you are not connecting from a CCRT partner network, you need to specify the external IP address from which you will access the machine. Any attempt to connect to the machine from another IP address will fail.</li>
</ul>
<p>Hereafter more information on how to connect to the supercomputer.</p>
<p>The supercomputer can be reached with the Secure SHell (SSH) protocol.</p>
<p>SSH connection to the supercomputer can be established:</p>
<ul class="simple">
<li>on Unix-like OSes (for instance Linux or Mac OS X) with the <strong class="command">ssh</strong> command within a terminal,</li>
<li>on Windows, with <strong class="program">PuTTY</strong> (we advise you to use version >= 0.59 for compatibility).The SSH client will open a terminal on a login node of the supercomputer.</li>
</ul>
<p>The SSH client will open a terminal on a login node of the supercomputer.</p>
<p>Access point depend of the type of node you want to access :</p>
<div class="section" id="skylake-fat-and-hybrid-nodes">
<h4>Skylake, Fat and Hybrid nodes<a class="headerlink" href="#skylake-fat-and-hybrid-nodes" title="Permalink to this headline">¶</a></h4>
<div class="admonition warning">
<p class="first admonition-title">Warning</p>
<p class="last">when using this access you wont be able to see any reference to rome or v100 nodes including jobs running on it.
To see Rome and V100 use connection method described in the next section.</p>
</div>
<p>If you own a PRACE user account (authentication with a password):</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ ssh <login>@irene-eu.ccc.cea.fr
password:
</pre></div>
</div>
<p>If you own a national academic (FR) user account (authentication with a password):</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ ssh <login>@irene-fr.ccc.cea.fr
password:
</pre></div>
</div>
<p>If you own a CCRT user account (authentication with a password or a SSH key-pair if you connect from a CCRT partner network):</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ ssh <login>@irene-ccrt.ccc.cea.fr
password:
</pre></div>
</div>
<p>To log out, you can use the Ctrl-d keyboard sequence, or type the <strong class="command">exit</strong> command.</p>
</div>
<div class="section" id="rome-and-v100">
<h4>Rome and V100<a class="headerlink" href="#rome-and-v100" title="Permalink to this headline">¶</a></h4>
<div class="admonition warning">
<p class="first admonition-title">Warning</p>
<p class="last">when using this access you wont be able to see any reference to skylake, knl, xlarge or hybrid nodes including jobs running on it.
To see skylake, knl, xlarge and hybrid use connection method described in the previous section.
If you own a PRACE user account (authentication with a password):</p>
</div>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ ssh <login>@irene-amd-eu.ccc.cea.fr
password:
</pre></div>
</div>
<p>If you own a national academic (FR) user account (authentication with a password):</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ ssh <login>@irene-amd-fr.ccc.cea.fr
password:
</pre></div>
</div>
<p>If you own a CCRT user account (authentication with a password or a SSH key-pair if you connect from a CCRT partner network):</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ ssh <login>@irene-amd-ccrt.ccc.cea.fr
password:
</pre></div>
</div>
<p>To log out, you can use the Ctrl-d keyboard sequence, or type the exit command.
Please note that:</p>
<ul class="simple">
<li>Idle connections will be automatically closed after 6 hours of inactivity.</li>
<li>You can only access the supercomputer from your local machine. By default, your local machine is not reachable from the supercomputer to increase the safety of your local environment.</li>
</ul>
<p>Limitation on the memory consumption on the login nodes: A user has a maximum of 8G RAM memory on a login node and all users have a limitation of 120G cumulated memory. If a program is executed on the login node and writes on /dev/shm it will also consume from the quota of 8G. Beyond this threshold a protection system will kill all the running processes including the connection processes, thus closing all user opened shells on the cluster.</p>
</div>
<div class="section" id="ssh-fingerprint">
<h4>SSH fingerprint<a class="headerlink" href="#ssh-fingerprint" title="Permalink to this headline">¶</a></h4>
<p>If a strict host key checking setting is enabled in your local OpenSSH configuration, you will have to register our system SSH fingerprint before your first connection:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ssh-keyscan irene.ccc.cea.fr >>~/.ssh/known_hosts
</pre></div>
</div>
<p>You can verify the SSH fingerprint registered for the system in your local OpenSSH configuration with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ssh-keygen -l -F irene.ccc.cea.fr
</pre></div>
</div>
<p>If we advertise a SSH fingerprint change, you can revoke previously registered fingerprint in your local OpenSSH configuration with either commands:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ssh-keygen -R irene.ccc.cea.fr
$ sed -i '/irene.ccc.cea.fr/d' ~/.ssh/known_hosts
</pre></div>
</div>
<p>To make you sure that the system you access is authentic, the SSH fingerprint presented to you should be:</p>
<ul class="simple">
<li>ECDSA key for irene.ccc.cea.fr, irene-fr.ccc.cea.fr, irene-ccrt.ccc.cea.fr, irene-ccrt.ccc.cea.fr or irene-eu.ccc.cea.fr:</li>
</ul>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>e7:2c:bb:c8:c3:ab:c3:9f:52:d5:50:a2:de:9a:2b:a7 <span class="o">(</span>MD5<span class="o">)</span>
knJaymnaa0dOL9GosYKfqntMs/zahoRSisiYLQz6WxY <span class="o">(</span>SHA256<span class="o">)</span>
</pre></div>
</div>
<ul class="simple">
<li>RSA key for irene.ccc.cea.fr or irene-fr.ccc.cea.fr:</li>
</ul>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="m">80</span>:5b:bf:1b:26:1b:b5:55:b3:fe:b6:ae:2b:05:09:f8 <span class="o">(</span>MD5<span class="o">)</span>
ZXohIopQqdipjXgZzladjpw8bg5Lxa1ilU1qSqqSRHQ <span class="o">(</span>SHA256<span class="o">)</span>
</pre></div>
</div>
<ul class="simple">
<li>RSA key for irene-ccrt.ccc.cea.fr:</li>
</ul>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>d5:33:c8:06:a6:54:7f:6a:ce:f8:0e:6d:36:f7:e4:e3 <span class="o">(</span>MD5<span class="o">)</span>
D5Cnp4HXnqg4nPPAwraKIpL/wXudnzS6a8XZW7FzA4c <span class="o">(</span>SHA256<span class="o">)</span>
</pre></div>
</div>
<ul class="simple">
<li>RSA key for irene-eu.ccc.cea.fr:</li>
</ul>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>3e:c4:0d:3b:c1:ad:df:b6:db:50:82:5d:7b:3f:bd:48 <span class="o">(</span>MD5<span class="o">)</span>
7WbUFzLzhX9zM2WLMq+RsuxU2Sji0dzORxR5O6TePB4 <span class="o">(</span>SHA256<span class="o">)</span>
</pre></div>
</div>
</div>
<div class="section" id="x11-connections">
<h4>X11 connections<a class="headerlink" href="#x11-connections" title="Permalink to this headline">¶</a></h4>
<p>If you need a graphical environment, you must forward the graphical protocol (X11) on your SSH connection. SSH provides the options <code class="xref std std-option docutils literal notranslate"><span class="pre">-X</span></code> and <code class="xref std std-option docutils literal notranslate"><span class="pre">-Y</span></code> to enable X11 protocol forwarding. For instance:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ssh -X <login>@irene.ccc.cea.fr
</pre></div>
</div>
<p>After 10 hours, you will need to renew your security privileges if you want to keep using graphical resources, by using the <strong class="command">kinit -R</strong> command.</p>
<p>Note that X11 forwarding should be enabled with caution. Connecting with X11 forwarding opens a channel from the server back to your graphical desktop environment. It means that if system is compromised, someone may access your local environment and spy on your activities.</p>
<p>Since this is a security issue, by default, X11 forwarding enabled with <code class="xref std std-option docutils literal notranslate"><span class="pre">-X</span></code> option is subjected to some restrictions by the system. This may cause some graphical software to fail. If this is the case, try connecting with <code class="xref std std-option docutils literal notranslate"><span class="pre">-Y</span></code>.</p>
<p>By using <code class="xref std std-option docutils literal notranslate"><span class="pre">-Y</span></code> instead of <code class="xref std std-option docutils literal notranslate"><span class="pre">-X</span></code>, the server is considered as “trusted”. The restrictions are then disabled so this mode should be used with particular caution.</p>
<p>Please refer to SSH man-pages for details.</p>
</div>
<div class="section" id="registering-an-external-host-to-connect-to-the-computing-center">
<h4>Registering an external host to connect to the computing center<a class="headerlink" href="#registering-an-external-host-to-connect-to-the-computing-center" title="Permalink to this headline">¶</a></h4>
<p>If you are not connecting from a CCRT partner network, you need to register the external host from which you will access the computing center. To declare an external host, you need to provide the following details to the TGCC Hotline:</p>
<ul class="simple">
<li>the <strong>IP address</strong> of the external host you will connect from; this IP address should be declared in a public DNS and should point to a public FQDN</li>
<li>the <strong>Fully Qualified Domain Name</strong> (FQDN) associated to the IP address; the external host you declare must have a public FQDN associated to its IP address</li>
<li>the <strong>contact information of your Security Officer</strong>; This person should guarantee that the configuration of the external host you register complies with the latest computer security practices and norms</li>
</ul>
<p>The above information have to be transmitted to register an external host to connect to the computing center. Your Security Officer and your Project Leader have to validate by mail to the TGCC Hotline your request.</p>
</div>
<div class="section" id="ssh-public-key-management">
<h4>SSH public key management<a class="headerlink" href="#ssh-public-key-management" title="Permalink to this headline">¶</a></h4>
<p><strong>For users connecting through a CCRT partners network</strong>, the security policy was enforced to use public-key cryptography. Key pairs will now have a validity period of one year and each user will be allowed a maximum number of five public keys.</p>
<p>Key pairs are still managed through the <code class="file docutils literal notranslate"><span class="pre">.ssh/authorized_keys</span></code> file. However, any change must now be followed by a synchronization request by email to the TGCC Hotline for immediate recognition. This synchronization will allow the new keys to be validated and registered in the authentication database used for the SSH access points.</p>
<div class="section" id="add-an-ssh-public-key">
<h5>Add an SSH public key<a class="headerlink" href="#add-an-ssh-public-key" title="Permalink to this headline">¶</a></h5>
<p>Any user connecting from a CCRT partners network and willing to connect to the supercomputer with an SSH public key should:</p>
<ul class="simple">
<li>add the specified SSH public key to the <code class="file docutils literal notranslate"><span class="pre">~/.ssh/authorized_keys</span></code> file on his CCRT account</li>
<li>request a synchronization of the <code class="file docutils literal notranslate"><span class="pre">.authorized_keys</span></code> file by sending an email to <a class="reference external" href="mailto:hotline.tgcc%40cea.fr">hotline<span>.</span>tgcc<span>@</span>cea<span>.</span>fr</a></li>
</ul>
<p>Here is an example of the different steps to follow. All these steps are executed on the user’s local workstation.</p>
<ul class="simple">
<li>generate a key pair for connection to irene</li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ssh-keygen -b 4096 -f ~/.ssh/id_rsa_ccrt
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ~/.ssh/id_rsa_ccrt.
Your public key has been saved in ~/.ssh/id_rsa_ccrt.pub.
The key fingerprint is:
55:f9:8b:42:99:ec:a8:6c:1d:48:35:73:de:2f:e5:f9 login@myhost
The key's randomart image is:
+--[ RSA 4096]----+
| .. |
| + ... |
| . *.+ . |
| . .* . o |
| . .S+ = o |
| . o o o = |
| . o . . . . |
| + . E|
| . |
+-----------------+
</pre></div>
</div>
<ul class="simple">
<li>associate the key pair to the irene connection in your <code class="file docutils literal notranslate"><span class="pre">~/.ssh/config</span></code> on your local workstation</li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ vim .ssh/config
$ grep -A7 irene .ssh/config
Host irene*
User login
IdentityFile ~/.ssh/id_rsa_ccrt
IdentitiesOnly yes
PubKeyAuthentication yes
PasswordAuthentication yes
GSSAPIAuthentication no
</pre></div>
</div>
<ul class="simple">
<li>add the public key to the <code class="file docutils literal notranslate"><span class="pre">authorized_keys</span></code> file in the user’s home directory</li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat ~/.ssh/id_rsa_ccrt.pub | ssh login@irene.ccc.cea.fr "cat - > ~/.ssh/authorized_keys"
login@irene.ccc.cea.fr's password:
</pre></div>
</div>
<ul class="simple">
<li>request the synchronization of the allowed public keys list for the user’s login. Here is a template of the mail to send to <a class="reference external" href="mailto:hotline.tgcc%40cea.fr">hotline<span>.</span>tgcc<span>@</span>cea<span>.</span>fr</a></li>
</ul>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Subject : Public Keys synchronization for user 'login'
> Dear TGCC Hotline,
>
> I have just modified my ~/.ssh/authorized_keys file. Please, synchronize its content to your authentication infrastructure.
>
> Thanks in advance.
> Regards,
</pre></div>
</div>
</div>
<div class="section" id="remove-an-ssh-public-key">
<h5>Remove an SSH public key<a class="headerlink" href="#remove-an-ssh-public-key" title="Permalink to this headline">¶</a></h5>
<p>Any user willing to remove an SSH public key from the list of authorized public keys should:</p>
<ul class="simple">
<li>remove the specified SSH public key from the <code class="file docutils literal notranslate"><span class="pre">~/.ssh/authorized_keys</span></code> file on his CCRT account</li>
<li>request a synchronization of the <code class="file docutils literal notranslate"><span class="pre">.authorized_keys</span></code> file by sending an email to <a class="reference external" href="mailto:hotline.tgcc%40cea.fr">hotline<span>.</span>tgcc<span>@</span>cea<span>.</span>fr</a></li>
</ul>
<p>Here is a template of the mail to send to <a class="reference external" href="mailto:hotline.tgcc%40cea.fr">hotline<span>.</span>tgcc<span>@</span>cea<span>.</span>fr</a></p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Subject : Public Keys synchronization for user 'login'
Dear TGCC Hotline,
>
> I have just modified my ~/.ssh/authorized_keys file. Please, synchronize its content to your authentication infrastructure.
>
> Thanks in advance.
> Regards,
</pre></div>
</div>
</div>
<div class="section" id="list-available-ssh-public-keys">
<h5>List available SSH public keys<a class="headerlink" href="#list-available-ssh-public-keys" title="Permalink to this headline">¶</a></h5>
<p>Listing all the available public keys associated to your account can be done from a login node by executing one of the following commands:</p>
<ul class="simple">
<li>list all valid public keys allowing to connect the supercomputer :</li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ssh-ldap-pubkey -u login -V valid list
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAulQnUFly4... login@mynhost expire=2016-03-31
</pre></div>
</div>
<ul class="simple">
<li>list all expired public keys that do not allow to connect to the supercomputer:</li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ssh-ldap-pubkey -u login -V expired list
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxoqrkZdj4... login@myphost expire=2015-01-26
</pre></div>
</div>
<ul class="simple">
<li>list all valid public keys that are due to expire within 30 days</li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ssh-ldap-pubkey -u login -V expire -e 30 list
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAulQnUFly... login@myhost expire=2015-04-25
</pre></div>
</div>
<div class="admonition warning">
<p class="first admonition-title">Warning</p>
<p class="last">Once a public key is removed by a user, it is saved in TGCC infrastructure for the record but does not allow to connect anymore.</p>
</div>
</div>
<div class="section" id="restore-the-ssh-authorized-keys-file">
<h5>Restore the .ssh/authorized_keys file<a class="headerlink" href="#restore-the-ssh-authorized-keys-file" title="Permalink to this headline">¶</a></h5>
<p>If your .ssh/authorized_keys is lost/corrupted, it can be restored thanks to the LDAP data with the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ssh-ldap-pubkey list -u login > ~/.ssh/authorized_keys
</pre></div>
</div>
<p>Once it has been restored, the authorized_keys file can be used again to add or remove public keys before any synchronisation request to the Hotline.</p>
</div>
</div>
<div class="section" id="login-node-load-balancing">
<h4>Login node load-balancing<a class="headerlink" href="#login-node-load-balancing" title="Permalink to this headline">¶</a></h4>
<p>When connecting to the supercomputer with its regular DNS name (irene.ccc.cea.fr), a user is automatically directed to one of the reachable login nodes based on the host he is coming from. So the subsequent connections from the same host will be directed to the same login node. If you want to connect to a specific login node, a specific DNS name has to be used:</p>
<ul class="simple">
<li>If you own a CCRT user account, these DNS names are:</li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">irene</span><span class="o">-</span><span class="n">log1</span><span class="o">.</span><span class="n">ccc</span><span class="o">.</span><span class="n">cea</span><span class="o">.</span><span class="n">fr</span>
<span class="n">irene</span><span class="o">-</span><span class="n">log2</span><span class="o">.</span><span class="n">ccc</span><span class="o">.</span><span class="n">cea</span><span class="o">.</span><span class="n">fr</span>
<span class="n">irene</span><span class="o">-</span><span class="n">log3</span><span class="o">.</span><span class="n">ccc</span><span class="o">.</span><span class="n">cea</span><span class="o">.</span><span class="n">fr</span>
</pre></div>
</div>
</div>
<div class="section" id="ssh-session-multiplexing">
<h4>SSH session multiplexing<a class="headerlink" href="#ssh-session-multiplexing" title="Permalink to this headline">¶</a></h4>
<p>SSH session multiplexing allows an existing connection to the computing center to be reused without re-authentication to the computing center. Anyone accessing the machine where the connection to the computing center is initiated can then reuse a multiplexed connection and access the computing center. This feature is available only when connecting from a CCRT partner network.</p>
</div>
</div>
<div class="section" id="login-nodes-usage">
<h3>Login nodes usage<a class="headerlink" href="#login-nodes-usage" title="Permalink to this headline">¶</a></h3>
<p>When you connect to the supercomputer, you are directed to one of the login nodes of the machine. Many users are simultaneously connected to these login nodes. Since their resources are shared it is important to respect some standards of good practice.</p>
<div class="section" id="usage-limit-on-login-nodes">
<h4>Usage limit on login nodes<a class="headerlink" href="#usage-limit-on-login-nodes" title="Permalink to this headline">¶</a></h4>
<p>Login nodes should only be used to interact with the batch manager and to run lightweight tasks. As a rule of thumb, any process or group of processes which would use more CPU power and/or memory than what is available on a basic personal computer should not be executed on a login node. For more demanding interactive tasks, you should allocate dedicated resources on a compute node.</p>
<p>To ensure a satisfying experience for all users, offending tasks are automatically throttled or killed.</p>
</div>
<div class="section" id="interactive-submission">
<h4>Interactive submission<a class="headerlink" href="#interactive-submission" title="Permalink to this headline">¶</a></h4>
<p>There are 2 possible ways of accessing computing resources without using a submission script. <strong class="command">ccc_mprun -K</strong> allows to create the allocation and the job environment while you are still on the login node. It is useful for MPI tests. <strong class="command">ccc_mprun -s</strong> opens a shell on a compute node. It is useful for sequential or multi-threaded work that would be too costly for the login node.</p>
<div class="section" id="allocate-resources-interactively-k">
<h5>Allocate resources interactively (-K)<a class="headerlink" href="#allocate-resources-interactively-k" title="Permalink to this headline">¶</a></h5>
<p>It is possible to work interactively on allocated compute nodes thanks to the option <code class="xref std std-option docutils literal notranslate"><span class="pre">-K</span></code> of <strong class="command">ccc_mprun</strong>.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ccc_mprun -p partition -n 8 -T 3600 -K
</pre></div>
</div>
<p>This command will create an allocation and start a job.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ echo $SLURM_JOBID
901732
</pre></div>
</div>
<p>Within this reservation, you can run job steps with <strong class="command">ccc_mprun</strong>. Since the compute nodes are already allocated, you will not have to wait.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ccc_mprun hostname
node1569
node1569
node1569
node1569
node1569
node1569
node1569
node1569
</pre></div>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">At this point, you are still connected on the login node. You cannot run your code directly or with mpirun. you have to use <strong class="command">ccc_mprun</strong>!</p>
</div>
<p>Such an allocation is useful when developing an MPI program, in order to be able to test a <strong class="command">ccc_mprun</strong> quickly and several times in a short period.</p>
</div>
<div class="section" id="working-on-a-compute-node-s">