How to programatically add attribute converter for specific entity fields?

Let’s say I have javax.persistence.AttributeConverter implementation called FooConverter and I want to apply this converter for specific entity fields.

If I would use the annotation approach it would look like:

@Column
@Convert(converter = FooConverter.class)
private String barField;

On orm XML mapping it would look like:

<entity class="com.example.FooBarEntity">
  <convert converter="com.example.FooConverter" attribute-name="barField"/>
</entity>

However, I would like to add this converter programmatically during javax.persistence.EntityManagerFactory configuration.

import org.hibernate.cfg.Configuration;
import org.springframework.orm.hibernate5.LocalSessionFactoryBuilder;

private EntityManagerFactory buildEntityManagerFactory() {
  Configuration config = new LocalSessionFactoryBuilder(dataSource);
  config.addAnnotatedClass(com.example.FooBarEntity.class);
  // I looking for something like config.addConverter(FooConverter, FooBarEntity, barField);
  return config.buildSessionFactory();
}

There is config.addAttributeConverter method but it seems it only creates converter instance and not bind it to the specific entity attribute. However, I want to apply the converter only for specific String fields so autoApply is not an option.