copying my comment from Slack over here: I like your ideas, in general. Additional label maps could fit into the bigger idea of having an arbitrary number of label maps, with a convention for specifying the label map name before the type label. These label map names could be created by the user, or by commands such as your psf reader. This functionality was the direction we were going in the original type labels PR, but we delayed this 'advanced' feature in favor of getting type labels implemented with fewer complications.

@alphataubio @jrgissing I disagree with (ab-)using label maps for storing generic properties. For that we have fix STORE/ATOM or fix property/atom, depending on whether users should have direct access to them or not.
Type labels should be restricted to types.

As for the read_psf command, I fail to see it creating atoms, bonds, angles, dihedrals, impropers, cmaps. Also there are several variants of PSF files with different formats (x-plor vs. charmm format and CMAP, CHEQ, and EXT variants). At the very least it should be flagged with an error, if an unsupported format is encountered.

As for the write_psf command, it is bad idea to use MPI_Allreduce() here. This needlessly blows up the memory requirements and also limits how large a system can be handled. Instead constructs should be used where MPI rank 0 receives chunks of "owned" data from the other ranks, and it should be investigated, if it is possible to back all topology data into a single communication buffer to avoid having to do N-1 communications for each item.

Since LAMMPS supports the polarizable CHARMM force field, an effort should be made to specifically import and write out such topologies.

As for the write_psf command, it is bad idea to use MPI_Allreduce() here. This needlessly blows up the memory requirements and also limits how large a system can be handled. Instead constructs should be used where MPI rank 0 receives chunks of "owned" data from the other ranks, and it should be investigated, if it is possible to back all topology data into a single communication buffer to avoid having to do N-1 communications for each item.

i copied that from src/write_data.cpp...

performance is not a big issue of concern here because you only call write_psf once to generate a psf for your system to import into a visualization program.

i copied that from src/write_data.cpp...

performance is not a big issue of concern here because you only call write_psf once to generate a psf for your system to import into a visualization program.

It is not about performance but about memory consumption, which directly impacts how large a system can be handled.
If the same construct exists in write_data, then this needs to be looked into as well. I'll talk this over with Steve at the next opportunity. We recently make some tests on the new machine in Oak Ridge and had to be "creative" to set up very large systems with using "replicate" multiple times.

@alphataubio @jrgissing I disagree with (ab-)using label maps for storing generic properties. For that we have fix STORE/ATOM or fix property/atom, depending on whether users should have direct access to them or not. Type labels should be restricted to types.

@akohlmey segment/residue/name are actually all kinds of types, and they need to be stored as integers in atom custom array with a map to strings, ie.

• selecting by segment, ie. protein (PROA) and ions (IONS) without solvent (SOLV)
• ARG for arginine, HIS for histidine, ..., TRP for tryptophan amino acids residues
• alpha carbon (CA) atoms by name
• ions types POT for potassium, SOD for sodium, CAL for calcium, MG for magnesium, CLA for chloride

labelmap atom 1 """ H """ 2 """ HA1 """ 3 """ HA2 """ 4 """ HA3 """ 5 """ HB1 """ 6 """ HB2 """ 7 """ HC """ 8 """ HP """ 9 """ HR1 """ 10 """ HR2 """ 11 """ HR3 """ 12 """ HS """ 14 """ C """ 15 """ CA """ 16 """ CAI """ 17 """ CC """ 18 """ CP1 """ 19 """ CP2 """ 20 """ CP3 """ 21 """ CPH1 """ 22 """ CPH2 """ 23 """ CPT """ 24 """ CT1 """ 25 """ CT2 """ 26 """ CT2A """ 27 """ CT3 """ 28 """ CY """ 29 """ N """ 30 """ NC2 """ 31 """ NH1 """ 32 """ NH2 """ 33 """ NH3 """ 34 """ NR1 """ 35 """ NR2 """ 36 """ NR3 """ 37 """ NY """ 38 """ O """ 39 """ OC """ 40 """ OH1 """ 44 """ S """ 45 """ CLA """ 47 """ CAL """
labelmap segment 1 """ PROA """ 2 """ SOLV """ 3 """ IONS """
labelmap residue 1 """ MET """ 2 """ ALA """ 3 """ GLU """ 4 """ LEU """ 5 """ VAL """ 6 """ ARG """ 7 """ CYS """ 8 """ ASP """ 9 """ THR """ 10 """ ASN """ 11 """ GLY """ 12 """ HSD """ 13 """ LYS """ 14 """ GLN """ 15 """ PRO """ 16 """ PHE """ 17 """ TRP """ 18 """ TYR """ 19 """ SER """ 20 """ ILE """ 21 """ HSP """ 22 """ CAL """ 23 """ CLA """
labelmap name 1 """ N """ 2 """ HT1 """ 3 """ HT2 """ 4 """ HT3 """ 5 """ CA """ 6 """ HA """ 7 """ CB """ 8 """ HB1 """ 9 """ HB2 """ 10 """ CG """ 11 """ HG1 """ 12 """ HG2 """ 13 """ SD """ 14 """ CE """ 15 """ HE1 """ 16 """ HE2 """ 17 """ HE3 """ 18 """ C """ 19 """ O """ 20 """ HN """ 21 """ HB3 """ 22 """ CD """ 23 """ OE1 """ 24 """ OE2 """ 25 """ HG """ 26 """ CD1 """ 27 """ HD11 """ 28 """ HD12 """ 29 """ HD13 """ 30 """ CD2 """ 31 """ HD21 """ 32 """ HD22 """ 33 """ HD23 """ 34 """ HB """ 35 """ CG1 """ 36 """ HG11 """ 37 """ HG12 """ 38 """ HG13 """ 39 """ CG2 """ 40 """ HG21 """ 41 """ HG22 """ 42 """ HG23 """ 43 """ HD1 """ 44 """ HD2 """ 45 """ NE """ 46 """ HE """ 47 """ CZ """ 48 """ NH1 """ 49 """ HH11 """ 50 """ HH12 """ 51 """ NH2 """ 52 """ HH21 """ 53 """ HH22 """ 54 """ SG """ 55 """ OD1 """ 56 """ OD2 """ 57 """ OG1 """ 58 """ ND2 """ 59 """ HA1 """ 60 """ HA2 """ 61 """ ND1 """ 62 """ CE1 """ 63 """ NE2 """ 64 """ NZ """ 65 """ HZ1 """ 66 """ HZ2 """ 67 """ HZ3 """ 68 """ HE21 """ 69 """ HE22 """ 70 """ HZ """ 71 """ CE2 """ 72 """ NE1 """ 73 """ CE3 """ 74 """ CZ3 """ 75 """ CZ2 """ 76 """ CH2 """ 77 """ HH2 """ 78 """ OH """ 79 """ HH """ 80 """ OG """ 81 """ HD3 """ 82 """ OT1 """ 83 """ OT2 """ 84 """ CAL """ 85 """ CLA """

As for the read_psf command, I fail to see it creating atoms, bonds, angles, dihedrals, impropers, cmaps.

it doesnt have to because that information is already correctly imported by read_data. Bond information in psf generated by charmmgui is wrong anyways. they have bonds between the two hydrogens in TIP3 molecules needed for shake to work in CHARMM MD ([BUG ?] bonds between TIP3 H1-H2 in Solution Builder) making TIP3 molecules look like triangles in licorice style for visualization:

The primary and only reason for read_psf is to read the segment/residue/name/type label information from a psf file. read_data reads the bonds properly, and then write_psf creates a psf for visualization without the triangle bonds for TIP3.

write_psf has the added benefit that it can export only a specific group instead of all, for example to visualize a solvated simulation without the explicit TIP3 molecules which cuts the trajectory files by >80% and speeds up rendering a lot.

Also, this PR now somewhat resolves https://matsci.org/t/following-water-box-in-lammps-guide-h2o-molecules-look-strange-in-dump-file/56123/9 because a previous read_psf is no longer required for write_psf to work. in that case, defaults are:

• numerical molecule id for segment/molecule/residue
• numerical type id for name/type

Also there are several variants of PSF files with different formats (x-plor vs. charmm format and CMAP, CHEQ, and EXT variants). At the very least it should be flagged with an error, if an unsupported format is encountered.

"The default PSF format of CHARMM is the same as "XPLOR" format. "EXT" is for "extended format", which allows longer names for atoms, residues, etc. "CMAP" is for dihedral cross-term corrections. "CHEQ" is an option which is not usually important, so we will not discuss it." https://www.charmm-gui.org/?doc=lecture&module=pdb&lesson=6

EXT format is 8 byte strings for segment/residue/name/type, backward compatible with non-EXT which is 4 byte strings. i'm using std::string and spliting on spaces with TextFileReader so it really doesnt matter. read_psf will work with bigger strings in the future without modifying my code.

"CHEQ is supported in the sense that CHEQ data is simply ignored." https://userguide.mdanalysis.org/1.1.1/formats/reference/psf.html#psf-spec

since everyone seems to be ignoring CHEQ, so will i. i can ignore CMAP also since that's getting imported by read_data and doesnt need to be exported by write_psf for visualization.

As for the write_psf command, it is bad idea to use MPI_Allreduce() here. This needlessly blows up the memory requirements and also limits how large a system can be handled. Instead constructs should be used where MPI rank 0 receives chunks of "owned" data from the other ranks, and it should be investigated, if it is possible to back all topology data into a single communication buffer to avoid having to do N-1 communications for each item.

ok ill rewrite it without MPI_Allreduce(), that's just what i started with since i copied from write_data. if it works well that can be a solution for write_data also.

Since LAMMPS supports the polarizable CHARMM force field, an effort should be made to specifically import and write out such topologies.

yes i started working with DRUDE recently for my journal article. CHARMM updated DRUDE parameters in 2023. However, there's no need to export DRUDE topology information to visualization programs AFAIK which is the primary purpose of this PR.

@akohlmey segment/residue/name are actually all kinds of types, and they need to be stored as integers in atom custom array with a map to strings, ie.

They are NOT force field types that are associated with force field parameters and that is for me the decisive point.
I strongly object to cluttering the core LAMMPS code with these kinds of "convenience" features.

If you want to implement something along those lines, It will have to be either done as a separate package without any significant changes to the core LAMMPS code, or you will have to fork LAMMPS into something called NAMMPS or LAMMD or similar and maintain your changes there and advertise it a "LAMMPS with NAMD support".

it doesnt have to because that information is already correctly imported by read_data

A read_psf command that does ignore most of the content of the file does not deserve to be called read_psf.
A proper read_psf command would replace read_data (when combined with reading the parameters and coordinates, although coordinates could also be read with read_dump).

The primary and only reason for read_psf is to read the segment/residue/name/type label information from a psf file. read_data reads the bonds properly, and then write_psf creates a psf for visualization without the triangle bonds for TIP3.

All of this can be easily done outside of LAMMPS.

write_psf has the added benefit that it can export only a specific group instead of all, for example to visualize a solvated simulation without the explicit TIP3 molecules which cuts the trajectory files by >80% and speeds up rendering a lot.

Again, that is not a concern of LAMMPS, but rather something that can very well be done outside of LAMMPS.

"The default PSF format of CHARMM is the same as "XPLOR" format. "EXT" is for "extended format", which allows longer names for atoms, residues, etc. "CMAP" is for dihedral cross-term corrections. "CHEQ" is an option which is not usually important, so we will not discuss it." https://www.charmm-gui.org/?doc=lecture&module=pdb&lesson=6

This is in contradiction with the psf plugin code in VMD and my personal recollection. CHARMM format has a number as atom type, while NAMD/X-PLOR uses a string.

@akohlmey @jrgissing @stanmoore1 PR4245 ready for review and merge

As already stated earlier and before looking at anything else, the changes to the labelmap code have to be removed. In my opinion, this is an undesired abuse of this facility for purposes that are not consistent the the rest, i.e. mapping types that are connected to styles that produce forces like pair, bond, angle, dihedral, and improper styles.

why would i reinvent the wheel from scratch when @jrgissing already did 90% of the work ? this was the original intent from the author of this class. you dont understand the universal applicability of this feature which is to map integers to strings and store string data for each atom (segment, residue, name, ...) efficiently in custom per-atom arrays.

should i copy the LabelMap class verbatim and call it StringMap ? then duplicate the functionality in variable.cpp by adding new functions that replicate exactly the functionality from labelmap but call it something different even though it does the same thing ?

why would i reinvent the wheel from scratch when @jrgissing already did 90% of the work ? this was the original intent from the author of this class. you dont understand the universal applicability of this feature which is to map integers to strings and store string data for each atom (segment, residue, name, ...) efficiently in custom per-atom arrays.

I see the convenience, but you are neglecting that from the maintainer perspective this is opening Pandora's box and will give everybody license to abuse this facility for their purposes. Right now, we have well defined boundaries, if we allow you to go beyond that, it will be difficult to argue with others and then we can end up with one big mess. Most of the added functionality here appears to be predominantly for your personal needs and workflows and has limited appeal to the general LAMMPS user community.

It was already a hard struggle to boil down the labelmap code to the essential parts and not have it blown up by functionality that wasn't essential or for which a specific use case did not exist yet.

There is a lot of redundancy in LAMMPS, just look at accelerator packages. With your line of arguing you could require, for example, that there should not be a KOKKOS package (or OPENMP or GPU or OPT) but the code folded directly into the core of LAMMPS. The OPENMP package was designed from the ground up to be a replacement for the core code and not a package. Making this an add-on package has caused quite a bit of complications and extra work, so you are not alone here.

So, as far as I am concerned, I would like to see that everything that are new features would be insulated from the core of LAMMPS and a seperate, optional package (call it CHARMM). There are precedents for that in the KIM or DRUDE or AMOEBA package, to name some examples.

should i copy the LabelMap class verbatim and call it StringMap ? then duplicate the functionality in variable.cpp by adding new functions that replicate exactly the functionality from labelmap but call it something different even though it does the same thing ?

I have not looked at the code at that level of detail, but just copying it would not be sufficient. The main goal would be to keep changes in an optional package and not add maintenance burden to the core code. That will likely need some alternate approaches to get the functionality you are after.

The way I see it, there are three different paths open to us, because I expect that we will have the same arguments with other of your pull requests, too:

1. We keep discussing and figure out together ways to modify your code so it can be a package that is optional and does require only, very minimal changes to the core LAMMPS code. This will be the most work (and probably frustration) for both sides, but should result in the best outcome for LAMMPS itself.
2. You let the core LAMMPS developers, i.e. mostly Stan and me, check out everything in your pull requests and we pick and choose what we see as useful for LAMMPS and what can be included without too much work and without too much clutter added. We most certainly will want to extract the bugfixes, but even there we may want to address them differently. Then it is your choice to decide to make changes for the rest and resubmit or only keep the changes in your repository only. This will be less effort for either side, but LAMMPS itself not have all the functionality.
3. You keep things as they are and then you maintain a fork of LAMMPS under a different name. We will be more than happy to have pointers to the fork on the LAMMPS homepage and possibly in the manual, pointing out where the particular benefits are. There is for example LIGGGHTS that came into existence this same way and has diverged from the current LAMMPS distribution quite significantly over time and come into its own. While this kind of split is not something we are looking for, it is a viable alternative, that gives you more freedom and avoids clashes. This is the least effort for either side, but also the least desired solution.

Let me emphasize at this point that we very much appreciate your efforts and your desire to contribute your changes back to LAMMPS. This is very commendable, since there are lots of people that don't do this and thus there are useful changes that do not make it into the distribution, which is sad. I very much understand that it is not always easy to deal with this. I am talking from personal experience, since I have been in your position and had to argue hard and regularly fail to get functionality into LAMMPS unchanged, where I personally thought the benefits should be obvious.

In almost all cases - after the initial frustration and anger - I came back with the requested changes or an alternative implementation and usually this was for the better.

At the same time, as a maintainer of LAMMPS I have the obligation to be this hard nosed for any changes to the core code. I can tell you that even now I experience these issues myself (the most recent example is the change to the Error class all() and one() functions which went through many iterations), you just don't see it from the outside. Sometimes these internal discussions are quite intense. Sometimes we do not find a good solution and end up with a compromise that makes neither "side" happy.

Hi @alphataubio @jrgissing @akohlmey

First, thanks @alphataubio for working on this idea to extend LAMMPS
with features useful for CHARMM-related modeling.

Second, I have not looked at your code or followed this discussion in great detail.

But I have a code-organization suggestion which might satisfy the concerns of @akohlmey
(to not change/complicate type labels) and provide a path forward to include a
version of your contribution in LAMMPS.

As I understand it, the new functionality needed for CHARMM residues,
etc, is to enable mapping of integers <--> string labels for arbitrary
user-defined per-atom properties, not core LAMMPS types (atom types,
bond types, etc). I can see that would be a convenient capability for
some use cases, including future features none of us have probably
thought about before.

My suggestion is to structure this in one of two ways. I will call
these (4,5) to append to the (1,2,3) list Axel made. But possibly
they are what Axel was thinking of as his part of his (1).

(4) Create a new AtomLabel class (just a suggested name) which derives
from the existing TypeLabel class (suggested rename of label_map.cpp).
Change nothing in the current TypeLabel if possible , other than
allowing it to be a parent class. Override any methods or add new
ones in AtomLabel, as needed for your new functionality. If, as you
say, atom labels are leveraging 90% of the type label functionality,
it seems like this re-organization would be fairly straitforward.

(5) Similar to (4) but create a new parent class IntLabel (suggested
name) which both TypeLabel and AtomLabel derive from. IntLabel has
data structs and methods common to both the Type and Atom variants.
Other classes only create TypeLatel or AtomLabel instances, not
IntLabel. Again, the goal is to change the existing code for
IntLabel + TypeLabel as little as possible.

The 2 or 3 resulting classes could be part of the LAMMPS core in
lammps/src. But I like the suggestion to put your other new features
for PSF/SDF commands in a CHARMM package, i.e. src/CHARMM. Other
CHARMM-related commands could be moved there as well: e.g. pair styles
and fix cmap.

I imagine Jake would be willing to help design this new int <--> label
class structure to generalize his current implementation.

This also allow future features to leverage this capability without
mucking with the existing classes. If needed, they can just derive
a new FooLabel class with any peculiar new features they need.

I'll add that there are many current features in LAMMPS which followed
this development path. Initially a class was added to implement a
one-off idea. Later, the developers or other contributors realized it
could be generalized. So the original class was re-organized to allow
for derived classes, or it was broken into an invisible parent class
with easy-to-add children.

Examples:
dump custom has childen CFG, NetCDF, VTK, ADIOS, etc
fix wall (invisible parent) has half a dozen children

HTH,
Steve

@akohlmey i dont need your workaround #4492 anymore, you can close it.

i added segment/residue/name to fix property/atom instead, it works great. see commits 318c049 to 97f4d96.

now i can from .restart of any timestep:

• import PSF data with new version of read_psf that doesnt use custom per-atom vectors. on the plus side, no more label maps needed either.
• delete water atoms (1M) to leave only proteins atoms and ions (~10000).
• export to PDB with write_pdb with segment/residue/name

and i swept all my mess into a new CHARMM package as requested/suggested by you and @sjplimp

READY FOR REVIEW