Results return slightly wrong date (sometimes)

Hibernate Search
1

Sometimes, when I search by date, it returns a date that is off by one day. This doesn’t happen often, but I have a test that demonstrates the issue. Can you tell me what I’m doing wrong?
I will share the test shortly.

2

Here is the test using Hibernate Search 5: https://drive.google.com/file/d/1-AjxBIuiPKgj3baSbt8lJYnEV7DsbkG3/view?usp=sharing

3

Nevermind. I think I solved it.

I added

.withZoneUTC()

to my createDate() class and it now seems to work as intended:

	public static Date createDate(String date) throws Exception {
		DateTimeFormatter format = DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss.SSS").withZoneUTC();

		Date parsedDate = format.parseDateTime(date).toDate();

		Calendar calendar = Calendar.getInstance();
		calendar.setTime(parsedDate);
		
		return calendar.getTime();
	}
4

Just realized that it can be simplified to just this:

	public static Date createDate(String date) throws Exception {
		DateTimeFormatter format = DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss.SSS").withZoneUTC();

		return format.parseDateTime(date).toDate();
	}
5

So, essentially the problem is here:

@DateBridge(resolution = Resolution.DAY, encoding=EncodingType.STRING)

In order to truncate a date/time to the day, you have to pick a timezone, because at a given instant, the day is different depending on the timezone: it may be Monday in Los Angeles, but already Tuesday in Tokyo.

For… “historical” reasons, the date bridge always picks UTC to perform the truncation. On the other hand, as you noticed, dates will, by default, use the default JVM timezone, which could be anything.

If, for the date you are indexing, the current day is different in UTC and in your default JVM timezone, then you’ll end up with the problem you mentioned.

This kind of problems is exactly why I’m reluctant to port this “feature” to Hibernate Search 6: it looks like it’s simple and works out of the box, but it’s not and it does not, and most of the time you’ll only notice something is off after you put an application in production.

Really, a better solution when you want to match all dates in a given day is to give up on this “resolution” feature (index hours, minutes, seconds as well) and to use a range query: use the start of the day (midnight) as the minimum, and the end of the day (11:59 PM) as the maximum. Then you’ll get every document whose date is sometime that day.

In most cases performance will be virtually identical, and you get much more flexibility (search for the past two weeks, a given year, an arbitrary range of days, a range of hours, …).

6

Thanks for the explanation @yrodiere. I hadn’t considered DateBridge as being part of the problem. When you say you’re reluctant to port the “resolution” feature to Hibernate 6, you’re still keeping the DateBridge, yes?
If you eliminate “resolution”, what date string value will be stored in the Lucene index? The exact one being passed in?
Perhaps it could be better to deprecate the resolution feature, for one version, to give folks an opportunity to migrate without breaking their apps.

7

Yes. The resolution parameter is gone, but Dates and Calendars can of course still be indexed.

It’s indexed as a long, but yes, we index the date that was passed in with full resolution down to the millisecond. More exactly we convert it to an Instant (date/time in the UTC timezone) this way, then we convert the Instant to a long this way. The same is done when building search queries, so the conversion to UTC will be transparent. More information about special cases and why using the legacy date/time API can be problematic here.

I wish that kind of things were enough, but to be honest there are so many changes in Hibernate Search 6 (including a new API) that large applications will require significant work to migrate, even with prior warnings.

However:

  • These changes happen for a reason, i.e. many nice features, including a more convenient DSL and first-class support for Elasticsearch. More information here.
  • There is already a minimal “compatibility layer” that should help with migrating (though it’s not intended for production). It will be in the next release of Hibernate Search 6. The migration layer includes (temporary) support for this “resolution” feature.
  • I’ll be working on a migration guide in the following weeks, it should give a better idea of what changed exactly.