[shivadarshan-devadiga]

Rationale for this change

For a Parquet file without an Avro schema in its metadata, such as files written by Spark, PyArrow or DuckDB, parquet-cli derives one with AvroSchemaConverter. That converter defaults to parquet.avro.add-list-element-records=true, which treats the single-field repeated group of a standard 3-level list as the element:

optional group tags (LIST) {
  repeated group list {
    optional binary element (STRING);
  }
}

So tags becomes an array of records list { element } instead of an array of strings. The Parquet spec's backward-compatibility rules for LIST treat such a group as the element only when it is named array or <list name>_tuple.

On master this breaks every command that uses the derived schema for such files:

• schema prints the record wrapper.
• cat, head, to-avro and convert fail with The requested schema is not compatible with the file schema. incompatible types: required group element { optional binary element (STRING); } != optional binary element (STRING). In 1.15.0, as reported in parquet-cli reports nested columns as null #3095, cat printed null for those columns instead.

What changes are included in this PR?

Schemas.fromParquet derives the Avro schema with add-list-element-records=false when the file has no Avro schema, so the converter follows the spec's LIST rules. If the user sets parquet.avro.add-list-element-records, that setting is kept. Files with an Avro schema in their metadata still use it, as before.

The open PR #3812 (GH-3672) would move cat, head and scan to the native Parquet reader, which would also avoid this failure for those commands. This change is independent of it: schema, to-avro and convert would keep using the derived Avro schema either way.

Scope: this fixes standard 3-level lists named list / element, the shape the spec recommends and that current writers produce. For other list shapes, schema now prints an array too, but reading with the derived schema as projection still returns null for the list. These are 2-level lists (repeated binary element), single-field groups with other names (for example bag or older PyArrow's list / item), and files that mix a standard list with one of these. That happens when the derived schema is used as the read projection, which this PR doesn't change. list / item and mixed files already returned null before this change. I'll follow up on those separately.

Are these changes tested?

Yes. ReadCliTest.catReadsStandardListsWithoutAvroSchema writes a 3-level list<string> with ExampleParquetWriter, so without an Avro schema, with a two-element list, an empty list, a null list and a list with a null element, and checks the cat output. SchemaCliTest.printsStandardListsWithoutAvroSchemaAsArrays checks that schema prints the list as an array of nullable strings. Both fail without this change. SchemaCliTest.keepsExplicitAddListElementRecordsSetting checks that -D parquet.avro.add-list-element-records=true still gives the record wrapper. All other parquet-cli tests pass.

I also checked a file written by PyArrow 26.0.1 with two list<string> columns: cat, head, schema, to-avro and convert now give the values PyArrow reads.

Are there any user-facing changes?

Yes. For Parquet files without an Avro schema, parquet-cli now reads and prints standard 3-level lists as arrays of their elements.

Closes #3095

[shivadarshan-devadiga]

@wgtmac Could you please approve the workflows and take a look when you have a moment?