Virtual machines to run EUTelescope analysis code accessing Test beam EOS data
Create the multipart file to be used for the user configuration:
write-mime-multipart -o user_data_context_mlt.txt user_data_ci.txt user_data_bs.txtTemplate file cmsit_tb_group_stack.yaml for orchestration:
heat_template_version: 2018-03-02
description: >
Create VM instances servers for CMS Inner Tracker Phase-II
test beam analysis
resources:
group:
type: OS::Heat::ResourceGroup
properties:
count: 10
resource_def:
type: OS::Nova::Server
properties:
name: "cmsit-tba0%index%"
key_name: lxplus
image: 65bc0185-7fba-4b08-b256-cfc7576d9dda
flavor: m2.large
metadata: {"cern-services": "true",
"landb-description":"TEST-BEAM EUDET type dockerized analysis server",
"landb-os": "LINUX",
"landb-mainuser": "CMS-IT-TB-SPS"
}
user_data_format: RAW
user_data:
get_file: user_data_context_mlt.txtCreate the stack of servers (10) using the template in the file cmsit_tb_group_stack.yaml:
openstack --os-project-id c44e1040-5691-4ea5-881c-eb9a4b4d97e6 stack create -t cmsit_tb_group_stack.yaml cmsit-tbaA stack of servers named cmsit-tb0<ID>.cern.ch will be created. Note there is no mechanism to balance the access to the servers so far. (TODO: Incorporate the servers into the puppet configuration mechanism and integrate them into an automatic load balancer)
In order to be able to access in the VM machine you have to belong to the CMS-IT-TB-SPS
e-group. Using your NICE user:
ssh -oGSSAPIDelegateCredentials=no <your user>@cmsit-tb0<ID>.cern.chBeing <ID> any number between 0 and 9.
Note the -o option in order not to accept delegated kerberos credentials. This will avoid having
problems with the AFS filesystem within the docker analysis container.
Those machines are intented to be used in a similar way than the lxplus machines, that means you
will have access to your home AFS (which is your home here as well), several EOS directories
(including CMS-related and USER) and some CVMFS software (desy).
/afs: You have read and write permissions like inlxplus- Your home directory inside
cmsit-tb0<ID>is the same than inlxplus
- Your home directory inside
/eos: You have read and write permissions like inlxplus- Test beam data is in
/eos/cms/store/group/dpg_tracker_upgrade/BeamTestTelescope
- Test beam data is in
- Software is placed under
/sw/reposfolder (although you don't have write permissions)
Once logged into the cmsit-tb0<ID> machine, you can launch the docker container
with the needed code to perform the EUTelescope offline analysis.
docker-analysisThe above command will create a container and give you a terminal inside it. Marlin,
root and other offline analysis commands are available in there. Again, /afs and
/eos folders are available, as well.
The steering files from the EUTelescope processor can be found in the docker-container at:
/home/eudaquser/sps-tb-201806-eutel-cfg, but in read only mode
- You don't have access to AFS/EOS in the
cmsit-tba<ID>machines:- Be sure you have your kerberos credentials:
kinit <niceuser>@CERN.CH
- Be sure you have your kerberos credentials:
- You don't have access to EOS in the docker container:
- Be sure you have created your kerberos credentials, then try to
ls /eos/cmsin thecmsit-tba<ID>machine. If there is no problems, enter into the container and try again
- Be sure you have created your kerberos credentials, then try to
Every time any of the main codes and configuration elements are modified, the stack should be reinitialized. The easy, less error-prone way to do it, is by destroying and re-creating the stack. The main codes are:
- eudaq, with the relevant docker image at dockerhub:eudaq
- eutelescope, with the relevant docker image at dockerhub:eutelescope Note that before creating the new stack, the software code changes should be propagated to the dockerhub images.
And the main configuration elements:
- user_data_bs.txt: user configuration, blablah
- user_data_ci.txt: user configuration, blahblah
- cmsit_tb_group_stack: The description file to create the stack.
The list of commands to destroy and create again the stack are described
below. The openstack commands are available at lxplus-cloud.cern.ch.
$ openstack project list# First get the stack id
$ openstack --os-project-id <PROJECTID> stack list
# Then delete them
$ openstack --os-project-id <PROJECTID> stack delete <STACKID>Wait until the vm instances are actually destroyed and all the resources freed.
Before create the stack, and if needed, create the multipart file
# Create the multipart config file
write-mime-multipart -o user_data_context_mlt.txt user_data_ci.txt user_data_bs.txt
# Create the stack
openstack --os-project-id c44e1040-5691-4ea5-881c-eb9a4b4d97e6 stack create -t cmsit_tb_group_stack.yaml cmsit-tba- User update (until a more smart mechanism is introduced, puppet for instance):
# As superuser, first add the user
> cern-config-users --setup-user-accounts
# Update the docker group with the new user, just run in the superuser home:
> ./post-install.sh- Repositories updates within the
/sw/reposfile. Use theanalyseruser (no login):
# Just sudo /bin/bash first
[root] cd /sw/repos/<repo_to_update>
[root] su -s /bin/bash -c 'git pull' analyser- Analysis software: use of software developement versions.
The
docker-analysisalias creates a container defined at/sw/repos/dockerfiles-eutelescope/docker-compose.yml. The container uses the eudaq and eutelescope softwares from the pre-built image. In order to use explicitely eudaq software from the host.- Use the EUDAQ software from the host
$ cd /sw/repos/dockerfiles-eutelescope # To compile EUTElescope using EUDAQ from the host, and let the compiled objects at the host (which could be use afterwards $ docker-compose run --rm compile # To used the host compiled EUDAQ with the EUTelescope image $ docker-compose run --rm devcode # To used the host compiled EUDAQ with the EUTelescope image, privileged mode, able to be used with the TLU $ docker-compose run --rm devcode
- EUTelescope development. In order to develop and test the EUTelescope code in the container, several steps are needed to dump the repo to the host and then be able to write code (in the host), compile (in the container) and use it (in the container).
Add the bind volume in the# Edit the docker-compose.override.yml file to include the binding of the host directory to the container $ sudo /bin/bash [root] su -s /bin/bash -c 'vim docker-compose.override.yml' analyser
devcodesection of the yaml file:Get an previous eutelescope container or create a new one, without deleting it:- type: bind source: /sw/repos/Eutelescope target: /eudaq/ilcsoft/v01-19-02/Eutelescope/
Remember to edit the source files on the host using the# Back as regular user [root] exit $ docker container ps -a # If no container, create one and exit $ docker run devcode # Entering again as superuser and copy the container eutelescope software into the host [root] su -s /bin/bash -c 'docker cp 7c252d34dbf2:/eudaq/ilcsoft/v01-19-02/Eutelescope/ /sw/repos/' analyser # Exit superuser and lauch the devcode service: $ docker-compose run --rm devcode
analyseruser:You must use a different container to run (su -s /bin/bash -c 'vim master/processors/src/EUTelAPIXTbTrackTuple.cc' analyseranalysismodified with the binded mounted src folder) than to compile.
- Puppet-managed VM at CERN: (foreman) https://judy.cern.ch
http://configdocs.web.cern.ch/configdocs- DNS Load balancing (https://aiermis.cern.ch)
- Self-managed Virtual Machines:
https://clouddocs.web.cern.ch/clouddocs- Stacks: Collection of resources which can be created by the Orchestration service (Heat),
including Vms, networks, rules, ...
https://clouddocs.web.cern.ch/clouddocs/tutorial/create_a_stack.htmlhttps://clouddocs.web.cern.ch/clouddocs/orchestration/advanced_concepts.html
- Cluster of docker containers: Swarm
- Stacks: Collection of resources which can be created by the Orchestration service (Heat),
including Vms, networks, rules, ...