Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 1dd8f4b

Browse files
agibsoncccagibsonccc
authored
Upgrades flat buffers to most recent version (#10189)
* update flatbuffers to new 64 bit version * Add ADR * fix namespace usage after new generation * Add rest of updates for flatc generated classes.
1 parent 2f1d459 commit 1dd8f4b

File tree

123 files changed

+709
-9817
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

123 files changed

+709
-9817
lines changed

‎ADRs/0034 - FlatBuffers upgrade.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# ADR 0034: FlatBuffers Modernization
2+
3+
## Status
4+
5+
Implemented
6+
7+
Proposed by: Assistant (20-02-2025)
8+
Discussed with: Adam Gibson
9+
10+
## Context
11+
12+
The libnd4j library uses FlatBuffers for serialization of neural network graphs and related data structures. The current implementation uses FlatBuffers 1.12.0 syntax and conventions, particularly for handling sequences and arrays. With the upgrade to newer versions of FlatBuffers, we need to modernize our schema definitions and code generation.
13+
14+
The primary challenges include:
15+
16+
1. Migration from legacy Sequence/SequenceItem patterns to modern vector types
17+
2. Proper generation of Java files with correct package structures
18+
3. Integration with build system for consistent compilation
19+
4. Maintaining backward compatibility where possible
20+
5. Ensuring proper environment variable passing for code generation
21+
6. Proper schema namespace management
22+
23+
## Decision
24+
25+
We implement a modernized FlatBuffers integration that uses current vector syntax and build processes. This includes:
26+
27+
### Key Components
28+
- Schema files using modern vector syntax (`[Type]` instead of Sequence)
29+
- Direct build process integration in CMake
30+
- Explicit environment variable handling for flatc compiler
31+
- Namespace standardization across schema files
32+
33+
### Implementation Details
34+
1. Schemas use vector syntax for array types (e.g., `[FlatArray]` instead of `Sequence<FlatArray>`)
35+
2. CMake executes flatc compilation directly instead of using custom targets
36+
3. Build process ensures flatc is compiled before schema generation
37+
4. Java package structure matches expected nd4j-api layout
38+
5. Schema namespaces standardized to `graph` without `sd` prefix
39+
40+
### Schema Example
41+
```fbs
42+
namespace graph;
43+
44+
table SequenceItem {
45+
name:string;
46+
associated_variable:[FlatArray];
47+
}
48+
49+
table SequenceItemRoot {
50+
sequence_items:[SequenceItem];
51+
}
52+
53+
root_type SequenceItemRoot;
54+
```
55+
56+
### Build Integration
57+
```cmake
58+
execute_process(
59+
COMMAND ${CMAKE_COMMAND} -E env "FLATC_PATH=${FLATC_EXECUTABLE}"
60+
bash ${CMAKE_CURRENT_SOURCE_DIR}/flatc-generate.sh
61+
RESULT_VARIABLE FLATC_RESULT
62+
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
63+
)
64+
```
65+
66+
## Consequences
67+
68+
### Advantages
69+
1. Modern FlatBuffers Integration:
70+
- Cleaner schema definitions
71+
- Better type safety through vector syntax
72+
- More maintainable code generation
73+
- Consistent with current FlatBuffers best practices
74+
75+
2. Build System Integration:
76+
- Reliable flatc compilation
77+
- Proper environment variable handling
78+
- Direct process execution instead of custom targets
79+
- Better error handling and reporting
80+
81+
3. Code Organization:
82+
- Correct Java package structure
83+
- Standardized namespaces
84+
- Clear separation of generated code
85+
- Better integration with existing nd4j structure
86+
87+
### Disadvantages
88+
1. Implementation Requirements:
89+
- Need to update existing schema files
90+
- Must ensure build process compatibility
91+
- Potential for temporary build issues during transition
92+
93+
2. Migration Effort:
94+
- Updates needed for existing code using legacy patterns
95+
- Need to verify all schema files
96+
- Testing required for all serialization paths
97+
98+
## Technical Details
99+
100+
### Schema Location
101+
```
102+
/libnd4j/include/graph/scheme/*.fbs
103+
```
104+
105+
### Generated Code Location
106+
```
107+
/nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/graph/
108+
```
109+
110+
### Build Process
111+
1. CMake configures build
112+
2. flatc compiler is built
113+
3. Schema generation script runs
114+
4. Java files are copied to appropriate location
115+
116+
## Alternatives Considered
117+
118+
1. Custom Target Approach:
119+
- Pros: More traditional CMake integration
120+
- Cons: Less direct control over process, harder to debug
121+
122+
2. Manual File Generation:
123+
- Pros: Simpler build process
124+
- Cons: Error-prone, harder to maintain
125+
126+
3. Separate Build Step:
127+
- Pros: Cleaner separation of concerns
128+
- Cons: More complex build process, potential for synchronization issues

‎deeplearning4j/deeplearning4j-ui-parent/deeplearning4j-vertx/pom.xml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -40,10 +40,10 @@
4040

4141
<build>
4242
<plugins>
43-
<!-- <plugin>
43+
<plugin>
4444
<groupId>org.moditect</groupId>
4545
<artifactId>moditect-maven-plugin</artifactId>
46-
</plugin>-->
46+
</plugin>
4747
</plugins>
4848
</build>
4949

@@ -309,9 +309,9 @@
309309
</dependency>
310310
<!-- For SameDiff UI -->
311311
<dependency>
312-
<groupId>org.webjars.npm</groupId>
313-
<artifactId>flatbuffers</artifactId>
314-
<version>1.9.0</version>
312+
<groupId>org.nd4j</groupId>
313+
<artifactId>nd4j-web</artifactId>
314+
<version>${project.version}</version>
315315
</dependency>
316316
</dependencies>
317317

‎libnd4j/CMakeLists.txt‎

Lines changed: 51 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -641,7 +641,14 @@ if (${HELPERS_cudnn})
641641
endif()
642642
endif()
643643

644-
# Download and unpack flatbuffers at configure time
644+
# Before the flatbuffers download section
645+
if(DEFINED ENV{GENERATE_FLATC} OR DEFINED GENERATE_FLATC)
646+
set(FLATBUFFERS_BUILD_FLATC "ON" CACHE STRING "Enable flatc build" FORCE)
647+
else()
648+
set(FLATBUFFERS_BUILD_FLATC "OFF" CACHE STRING "Disable flatc build" FORCE)
649+
endif()
650+
651+
# Existing flatbuffers download section
645652
configure_file(CMakeLists.txt.in flatbuffers-download/CMakeLists.txt)
646653
execute_process(COMMAND ${CMAKE_COMMAND} "${CMAKE_GENERATOR}" .
647654
RESULT_VARIABLE result
@@ -656,15 +663,54 @@ if(result)
656663
message(FATAL_ERROR "Build step for flatbuffers failed: ${result}")
657664
endif()
658665

659-
# Add flatbuffers directly to our build.
666+
# Add flatbuffers directly to build
660667
add_subdirectory(${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src
661668
${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-build
662669
EXCLUDE_FROM_ALL)
663670

664-
set(HAVE_FLATBUFFERS 1)
665-
set(FLATBUFFERS_PATH ${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src)
666-
include_directories(${FLATBUFFERS_PATH}/include)
671+
# After flatbuffers is added, handle generation if enabled
672+
if(DEFINED ENV{GENERATE_FLATC} OR DEFINED GENERATE_FLATC)
673+
# First, ensure flatc is built
674+
execute_process(
675+
COMMAND ${CMAKE_COMMAND} --build ${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-build --target flatc
676+
RESULT_VARIABLE FLATC_BUILD_RESULT
677+
)
678+
if(FLATC_BUILD_RESULT)
679+
message(FATAL_ERROR "Failed to build flatc: ${FLATC_BUILD_RESULT}")
680+
endif()
681+
667682

683+
set(FLATC_EXECUTABLE "${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-build/flatc")
684+
message("Using flatc from: ${FLATC_EXECUTABLE}")
685+
686+
# Create required directories
687+
execute_process(COMMAND ${CMAKE_COMMAND} -E make_directory
688+
${CMAKE_CURRENT_SOURCE_DIR}/include/graph/generated
689+
${CMAKE_CURRENT_SOURCE_DIR}/include/graph/generated/sd
690+
)
691+
692+
# Run flatc generation
693+
execute_process(
694+
COMMAND ${CMAKE_COMMAND} -E env "FLATC_PATH=${FLATC_EXECUTABLE}"
695+
bash ${CMAKE_CURRENT_SOURCE_DIR}/flatc-generate.sh
696+
RESULT_VARIABLE FLATC_RESULT
697+
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
698+
)
699+
if(FLATC_RESULT)
700+
message(FATAL_ERROR "Flatbuffer generation failed: ${FLATC_RESULT}")
701+
endif()
702+
703+
# Copy Java files
704+
execute_process(
705+
COMMAND bash ${CMAKE_CURRENT_SOURCE_DIR}/copy-flatc-java.sh
706+
RESULT_VARIABLE COPY_RESULT
707+
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
708+
)
709+
if(COPY_RESULT)
710+
message(FATAL_ERROR "Java file copying failed: ${COPY_RESULT}")
711+
endif()
712+
endif()
713+
include_directories(${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src/include)
668714

669715
configure_file(include/config.h.in include/config.h)
670716
include_directories(${CMAKE_CURRENT_BINARY_DIR}/include)

‎libnd4j/CMakeLists.txt.in‎

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,22 @@ cmake_minimum_required(VERSION 2.8.2)
22

33
project(flatbuffers-download NONE)
44

5+
# Force clean everything first
6+
file(REMOVE_RECURSE ${CMAKE_CURRENT_BINARY_DIR}/*)
7+
58
include(ExternalProject)
69
ExternalProject_Add(flatbuffers
7-
GIT_REPOSITORY https://github.com/KonduitAI/flatbuffers.git
8-
GIT_TAG main
9-
SOURCE_DIR "${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src"
10-
BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-build"
11-
CONFIGURE_COMMAND ""
12-
CMAKE_ARGS "-DFLATBUFFERS_BUILD_FLATC=OFF"
13-
BUILD_COMMAND ""
14-
INSTALL_COMMAND ""
15-
TEST_COMMAND ""
16-
)
10+
GIT_REPOSITORY https://github.com/google/flatbuffers/
11+
GIT_TAG v25.2.10
12+
SOURCE_DIR "${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src"
13+
BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-build"
14+
CMAKE_ARGS
15+
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
16+
-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
17+
-DFLATBUFFERS_BUILD_FLATC=ON
18+
-DCMAKE_BUILD_TYPE=Release
19+
DOWNLOAD_COMMAND ${CMAKE_COMMAND} -E rm -rf ${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src && git clone https://github.com/google/flatbuffers/ ${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src
20+
UPDATE_COMMAND ""
21+
INSTALL_COMMAND ""
22+
TEST_COMMAND ""
23+
)

‎libnd4j/CMakePresets.json‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
"CMAKE_INSTALL_PREFIX": "${sourceDir}/out/install/${presetName}",
1313
"SD_LIBRARY_NAME": "nd4jcpu",
1414
"SD_CPU": true,
15+
"GENERATE_FLATC": "ON",
1516
"SD_ARCH": "x86-64",
1617
"SD_GCC_FUNCTRACE": "ON",
1718
"PRINT_MATH": "ON",
@@ -36,7 +37,7 @@
3637
"PRINT_INDICES": "ON",
3738
"SD_BUILD_TESTS": "OFF",
3839
"PRINT_MATH": "ON",
39-
40+
"GENERATE_FLATC": "ON",
4041
"SD_ALL_OPS": true,
4142
"CMAKE_BUILD_TYPE" : "Debug",
4243
"OPENBLAS_PATH": "$env{HOME}/.javacpp/cache/openblas-0.3.19-1.5.7-linux-x86_64.jar/org/bytedeco/openblas/linux-x86_64"
@@ -62,6 +63,7 @@
6263
"SD_GCC_FUNCTRACE": "ON",
6364
"CMAKE_CUDA_ARCHITECTURES": "86",
6465
"SD_BUILD_TESTS": "OFF",
66+
"GENERATE_FLATC": "ON",
6567
"CUDA_TOOLKIT_ROOT_DIR": "/usr/local/cuda",
6668
"CMAKE_CUDA_COMPILER": "/usr/local/cuda/bin/nvcc"
6769
}
@@ -80,6 +82,7 @@
8082
"SD_LIBRARY_NAME": "nd4jcuda",
8183
"SD_ALL_OPS": true,
8284
"SD_CUDA": true,
85+
"GENERATE_FLATC": "ON",
8386
"BLAS":true,
8487
"SD_GCC_FUNCTRACE": "ON",
8588
"__CUDACC__" : "ON",

‎libnd4j/buildnativeoperations.sh‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,13 +93,19 @@ LOG_OUTPUT="none"
9393
PRINT_MATH="OFF"
9494
KEEP_NVCC="OFF"
9595
PREPROCESS="ON" # Initialize PREPROCESS variable
96+
CMAKE_ARGUMENTS=""
9697

9798
while [[ $# -gt 0 ]]
9899
do
99100
key="$1"
100101
value="${2:-}"
101102
# Build type (release/debug), packaging type, chip: cpu,cuda, lib type (static/dynamic)
102103
case $key in
104+
--generate-flatc)
105+
export GENERATE_FLATC="ON"
106+
CMAKE_ARGUMENTS="$CMAKE_ARGUMENTS -DGENERATE_FLATC=ON"
107+
shift # past argument
108+
;;
103109
-ol|--optimization-level)
104110
OPTIMIZATION_LEVEL="$value"
105111
shift # past argument

‎libnd4j/cmake/aurora.cmake‎

Lines changed: 0 additions & 42 deletions
This file was deleted.

0 commit comments

Comments
 (0)