commit fdd8180
sewn
·
2026-10-25 13:20:39 +0000 UTC
parent 059a147
stb_image_resize2: upgrade to 2.50
1 files changed,
+13549,
-10679
+13549,
-10679
1@@ -1,10679 +1,13549 @@
2-/* stb_image_resize2 - v2.18 - public domain image resizing
3-
4- by Jeff Roberts (v2) and Jorge L Rodriguez
5- http://github.com/nothings/stb
6-
7- Can be threaded with the extended API. SSE2, AVX, Neon and WASM SIMD support. Only
8- scaling and translation is supported, no rotations or shears.
9-
10- COMPILING & LINKING
11- In one C/C++ file that #includes this file, do this:
12- #define STB_IMAGE_RESIZE_IMPLEMENTATION
13- before the #include. That will create the implementation in that file.
14-
15- EASY API CALLS:
16- Easy API downsamples w/Mitchell filter, upsamples w/cubic interpolation, clamps to edge.
17-
18- stbir_resize_uint8_srgb( input_pixels, input_w, input_h, input_stride_in_bytes,
19- output_pixels, output_w, output_h, output_stride_in_bytes,
20- pixel_layout_enum )
21-
22- stbir_resize_uint8_linear( input_pixels, input_w, input_h, input_stride_in_bytes,
23- output_pixels, output_w, output_h, output_stride_in_bytes,
24- pixel_layout_enum )
25-
26- stbir_resize_float_linear( input_pixels, input_w, input_h, input_stride_in_bytes,
27- output_pixels, output_w, output_h, output_stride_in_bytes,
28- pixel_layout_enum )
29-
30- If you pass NULL or zero for the output_pixels, we will allocate the output buffer
31- for you and return it from the function (free with free() or STBIR_FREE).
32- As a special case, XX_stride_in_bytes of 0 means packed continuously in memory.
33-
34- API LEVELS
35- There are three levels of API - easy-to-use, medium-complexity and extended-complexity.
36-
37- See the "header file" section of the source for API documentation.
38-
39- ADDITIONAL DOCUMENTATION
40-
41- MEMORY ALLOCATION
42- By default, we use malloc and free for memory allocation. To override the
43- memory allocation, before the implementation #include, add a:
44-
45- #define STBIR_MALLOC(size,user_data) ...
46- #define STBIR_FREE(ptr,user_data) ...
47-
48- Each resize makes exactly one call to malloc/free (unless you use the
49- extended API where you can do one allocation for many resizes). Under
50- address sanitizer, we do separate allocations to find overread/writes.
51-
52- PERFORMANCE
53- This library was written with an emphasis on performance. When testing
54- stb_image_resize with RGBA, the fastest mode is STBIR_4CHANNEL with
55- STBIR_TYPE_UINT8 pixels and CLAMPed edges (which is what many other resize
56- libs do by default). Also, make sure SIMD is turned on of course (default
57- for 64-bit targets). Avoid WRAP edge mode if you want the fastest speed.
58-
59- This library also comes with profiling built-in. If you define STBIR_PROFILE,
60- you can use the advanced API and get low-level profiling information by
61- calling stbir_resize_extended_profile_info() or stbir_resize_split_profile_info()
62- after a resize.
63-
64- SIMD
65- Most of the routines have optimized SSE2, AVX, NEON and WASM versions.
66-
67- On Microsoft compilers, we automatically turn on SIMD for 64-bit x64 and
68- ARM; for 32-bit x86 and ARM, you select SIMD mode by defining STBIR_SSE2 or
69- STBIR_NEON. For AVX and AVX2, we auto-select it by detecting the /arch:AVX
70- or /arch:AVX2 switches. You can also always manually turn SSE2, AVX or AVX2
71- support on by defining STBIR_SSE2, STBIR_AVX or STBIR_AVX2.
72-
73- On Linux, SSE2 and Neon is on by default for 64-bit x64 or ARM64. For 32-bit,
74- we select x86 SIMD mode by whether you have -msse2, -mavx or -mavx2 enabled
75- on the command line. For 32-bit ARM, you must pass -mfpu=neon-vfpv4 for both
76- clang and GCC, but GCC also requires an additional -mfp16-format=ieee to
77- automatically enable NEON.
78-
79- On x86 platforms, you can also define STBIR_FP16C to turn on FP16C instructions
80- for converting back and forth to half-floats. This is autoselected when we
81- are using AVX2. Clang and GCC also require the -mf16c switch. ARM always uses
82- the built-in half float hardware NEON instructions.
83-
84- You can also tell us to use multiply-add instructions with STBIR_USE_FMA.
85- Because x86 doesn't always have fma, we turn it off by default to maintain
86- determinism across all platforms. If you don't care about non-FMA determinism
87- and are willing to restrict yourself to more recent x86 CPUs (around the AVX
88- timeframe), then fma will give you around a 15% speedup.
89-
90- You can force off SIMD in all cases by defining STBIR_NO_SIMD. You can turn
91- off AVX or AVX2 specifically with STBIR_NO_AVX or STBIR_NO_AVX2. AVX is 10%
92- to 40% faster, and AVX2 is generally another 12%.
93-
94- ALPHA CHANNEL
95- Most of the resizing functions provide the ability to control how the alpha
96- channel of an image is processed.
97-
98- When alpha represents transparency, it is important that when combining
99- colors with filtering, the pixels should not be treated equally; they
100- should use a weighted average based on their alpha values. For example,
101- if a pixel is 1% opaque bright green and another pixel is 99% opaque
102- black and you average them, the average will be 50% opaque, but the
103- unweighted average and will be a middling green color, while the weighted
104- average will be nearly black. This means the unweighted version introduced
105- green energy that didn't exist in the source image.
106-
107- (If you want to know why this makes sense, you can work out the math for
108- the following: consider what happens if you alpha composite a source image
109- over a fixed color and then average the output, vs. if you average the
110- source image pixels and then composite that over the same fixed color.
111- Only the weighted average produces the same result as the ground truth
112- composite-then-average result.)
113-
114- Therefore, it is in general best to "alpha weight" the pixels when applying
115- filters to them. This essentially means multiplying the colors by the alpha
116- values before combining them, and then dividing by the alpha value at the
117- end.
118-
119- The computer graphics industry introduced a technique called "premultiplied
120- alpha" or "associated alpha" in which image colors are stored in image files
121- already multiplied by their alpha. This saves some math when compositing,
122- and also avoids the need to divide by the alpha at the end (which is quite
123- inefficient). However, while premultiplied alpha is common in the movie CGI
124- industry, it is not commonplace in other industries like videogames, and most
125- consumer file formats are generally expected to contain not-premultiplied
126- colors. For example, Photoshop saves PNG files "unpremultiplied", and web
127- browsers like Chrome and Firefox expect PNG images to be unpremultiplied.
128-
129- Note that there are three possibilities that might describe your image
130- and resize expectation:
131-
132- 1. images are not premultiplied, alpha weighting is desired
133- 2. images are not premultiplied, alpha weighting is not desired
134- 3. images are premultiplied
135-
136- Both case #2 and case #3 require the exact same math: no alpha weighting
137- should be applied or removed. Only case 1 requires extra math operations;
138- the other two cases can be handled identically.
139-
140- stb_image_resize expects case #1 by default, applying alpha weighting to
141- images, expecting the input images to be unpremultiplied. This is what the
142- COLOR+ALPHA buffer types tell the resizer to do.
143-
144- When you use the pixel layouts STBIR_RGBA, STBIR_BGRA, STBIR_ARGB,
145- STBIR_ABGR, STBIR_RA, or STBIR_AR you are telling us that the pixels are
146- non-premultiplied. In these cases, the resizer will alpha weight the colors
147- (effectively creating the premultiplied image), do the filtering, and then
148- convert back to non-premult on exit.
149-
150- When you use the pixel layouts STBIR_RGBA_PM, STBIR_BGRA_PM, STBIR_ARGB_PM,
151- STBIR_ABGR_PM, STBIR_RA_PM or STBIR_AR_PM, you are telling that the pixels
152- ARE premultiplied. In this case, the resizer doesn't have to do the
153- premultipling - it can filter directly on the input. This about twice as
154- fast as the non-premultiplied case, so it's the right option if your data is
155- already setup correctly.
156-
157- When you use the pixel layout STBIR_4CHANNEL or STBIR_2CHANNEL, you are
158- telling us that there is no channel that represents transparency; it may be
159- RGB and some unrelated fourth channel that has been stored in the alpha
160- channel, but it is actually not alpha. No special processing will be
161- performed.
162-
163- The difference between the generic 4 or 2 channel layouts, and the
164- specialized _PM versions is with the _PM versions you are telling us that
165- the data *is* alpha, just don't premultiply it. That's important when
166- using SRGB pixel formats, we need to know where the alpha is, because
167- it is converted linearly (rather than with the SRGB converters).
168-
169- Because alpha weighting produces the same effect as premultiplying, you
170- even have the option with non-premultiplied inputs to let the resizer
171- produce a premultiplied output. Because the intially computed alpha-weighted
172- output image is effectively premultiplied, this is actually more performant
173- than the normal path which un-premultiplies the output image as a final step.
174-
175- Finally, when converting both in and out of non-premulitplied space (for
176- example, when using STBIR_RGBA), we go to somewhat heroic measures to
177- ensure that areas with zero alpha value pixels get something reasonable
178- in the RGB values. If you don't care about the RGB values of zero alpha
179- pixels, you can call the stbir_set_non_pm_alpha_speed_over_quality()
180- function - this runs a premultiplied resize about 25% faster. That said,
181- when you really care about speed, using premultiplied pixels for both in
182- and out (STBIR_RGBA_PM, etc) much faster than both of these premultiplied
183- options.
184-
185- PIXEL LAYOUT CONVERSION
186- The resizer can convert from some pixel layouts to others. When using the
187- stbir_set_pixel_layouts(), you can, for example, specify STBIR_RGBA
188- on input, and STBIR_ARGB on output, and it will re-organize the channels
189- during the resize. Currently, you can only convert between two pixel
190- layouts with the same number of channels.
191-
192- DETERMINISM
193- We commit to being deterministic (from x64 to ARM to scalar to SIMD, etc).
194- This requires compiling with fast-math off (using at least /fp:precise).
195- Also, you must turn off fp-contracting (which turns mult+adds into fmas)!
196- We attempt to do this with pragmas, but with Clang, you usually want to add
197- -ffp-contract=off to the command line as well.
198-
199- For 32-bit x86, you must use SSE and SSE2 codegen for determinism. That is,
200- if the scalar x87 unit gets used at all, we immediately lose determinism.
201- On Microsoft Visual Studio 2008 and earlier, from what we can tell there is
202- no way to be deterministic in 32-bit x86 (some x87 always leaks in, even
203- with fp:strict). On 32-bit x86 GCC, determinism requires both -msse2 and
204- -fpmath=sse.
205-
206- Note that we will not be deterministic with float data containing NaNs -
207- the NaNs will propagate differently on different SIMD and platforms.
208-
209- If you turn on STBIR_USE_FMA, then we will be deterministic with other
210- fma targets, but we will differ from non-fma targets (this is unavoidable,
211- because a fma isn't simply an add with a mult - it also introduces a
212- rounding difference compared to non-fma instruction sequences.
213-
214- FLOAT PIXEL FORMAT RANGE
215- Any range of values can be used for the non-alpha float data that you pass
216- in (0 to 1, -1 to 1, whatever). However, if you are inputting float values
217- but *outputting* bytes or shorts, you must use a range of 0 to 1 so that we
218- scale back properly. The alpha channel must also be 0 to 1 for any format
219- that does premultiplication prior to resizing.
220-
221- Note also that with float output, using filters with negative lobes, the
222- output filtered values might go slightly out of range. You can define
223- STBIR_FLOAT_LOW_CLAMP and/or STBIR_FLOAT_HIGH_CLAMP to specify the range
224- to clamp to on output, if that's important.
225-
226- MAX/MIN SCALE FACTORS
227- The input pixel resolutions are in integers, and we do the internal pointer
228- resolution in size_t sized integers. However, the scale ratio from input
229- resolution to output resolution is calculated in float form. This means
230- the effective possible scale ratio is limited to 24 bits (or 16 million
231- to 1). As you get close to the size of the float resolution (again, 16
232- million pixels wide or high), you might start seeing float inaccuracy
233- issues in general in the pipeline. If you have to do extreme resizes,
234- you can usually do this is multiple stages (using float intermediate
235- buffers).
236-
237- FLIPPED IMAGES
238- Stride is just the delta from one scanline to the next. This means you can
239- use a negative stride to handle inverted images (point to the final
240- scanline and use a negative stride). You can invert the input or output,
241- using negative strides.
242-
243- DEFAULT FILTERS
244- For functions which don't provide explicit control over what filters to
245- use, you can change the compile-time defaults with:
246-
247- #define STBIR_DEFAULT_FILTER_UPSAMPLE STBIR_FILTER_something
248- #define STBIR_DEFAULT_FILTER_DOWNSAMPLE STBIR_FILTER_something
249-
250- See stbir_filter in the header-file section for the list of filters.
251-
252- NEW FILTERS
253- A number of 1D filter kernels are supplied. For a list of supported
254- filters, see the stbir_filter enum. You can install your own filters by
255- using the stbir_set_filter_callbacks function.
256-
257- PROGRESS
258- For interactive use with slow resize operations, you can use the
259- scanline callbacks in the extended API. It would have to be a *very* large
260- image resample to need progress though - we're very fast.
261-
262- CEIL and FLOOR
263- In scalar mode, the only functions we use from math.h are ceilf and floorf,
264- but if you have your own versions, you can define the STBIR_CEILF(v) and
265- STBIR_FLOORF(v) macros and we'll use them instead. In SIMD, we just use
266- our own versions.
267-
268- ASSERT
269- Define STBIR_ASSERT(boolval) to override assert() and not use assert.h
270-
271- PORTING FROM VERSION 1
272- The API has changed. You can continue to use the old version of stb_image_resize.h,
273- which is available in the "deprecated/" directory.
274-
275- If you're using the old simple-to-use API, porting is straightforward.
276- (For more advanced APIs, read the documentation.)
277-
278- stbir_resize_uint8():
279- - call `stbir_resize_uint8_linear`, cast channel count to `stbir_pixel_layout`
280-
281- stbir_resize_float():
282- - call `stbir_resize_float_linear`, cast channel count to `stbir_pixel_layout`
283-
284- stbir_resize_uint8_srgb():
285- - function name is unchanged
286- - cast channel count to `stbir_pixel_layout`
287- - above is sufficient unless your image has alpha and it's not RGBA/BGRA
288- - in that case, follow the below instructions for stbir_resize_uint8_srgb_edgemode
289-
290- stbir_resize_uint8_srgb_edgemode()
291- - switch to the "medium complexity" API
292- - stbir_resize(), very similar API but a few more parameters:
293- - pixel_layout: cast channel count to `stbir_pixel_layout`
294- - data_type: STBIR_TYPE_UINT8_SRGB
295- - edge: unchanged (STBIR_EDGE_WRAP, etc.)
296- - filter: STBIR_FILTER_DEFAULT
297- - which channel is alpha is specified in stbir_pixel_layout, see enum for details
298-
299- FUTURE TODOS
300- * For polyphase integral filters, we just memcpy the coeffs to dupe
301- them, but we should indirect and use the same coeff memory.
302- * Add pixel layout conversions for sensible different channel counts
303- (maybe, 1->3/4, 3->4, 4->1, 3->1).
304- * For SIMD encode and decode scanline routines, do any pre-aligning
305- for bad input/output buffer alignments and pitch?
306- * For very wide scanlines, we should we do vertical strips to stay within
307- L2 cache. Maybe do chunks of 1K pixels at a time. There would be
308- some pixel reconversion, but probably dwarfed by things falling out
309- of cache. Probably also something possible with alternating between
310- scattering and gathering at high resize scales?
311- * Should we have a multiple MIPs at the same time function (could keep
312- more memory in cache during multiple resizes)?
313- * Rewrite the coefficient generator to do many at once.
314- * AVX-512 vertical kernels - worried about downclocking here.
315- * Convert the reincludes to macros when we know they aren't changing.
316- * Experiment with pivoting the horizontal and always using the
317- vertical filters (which are faster, but perhaps not enough to overcome
318- the pivot cost and the extra memory touches). Need to buffer the whole
319- image so have to balance memory use.
320- * Most of our code is internally function pointers, should we compile
321- all the SIMD stuff always and dynamically dispatch?
322-
323- CONTRIBUTORS
324- Jeff Roberts: 2.0 implementation, optimizations, SIMD
325- Martins Mozeiko: NEON simd, WASM simd, clang and GCC whisperer
326- Fabian Giesen: half float and srgb converters
327- Sean Barrett: API design, optimizations
328- Jorge L Rodriguez: Original 1.0 implementation
329- Aras Pranckevicius: bugfixes
330- Nathan Reed: warning fixes for 1.0
331-
332- REVISIONS
333- 2.18 (2026-03-25) fixed coefficient calculation when skipping a coefficient off
334- the left side of the window, added non-aligned access safe
335- memcpy mode for scalar path, fixed various typos, and fixed
336- define error in the float clamp output mode.
337- 2.17 (2025-10-25) silly format bug in easy-to-use APIs.
338- 2.16 (2025-10-21) fixed the easy-to-use APIs to allow inverted bitmaps (negative
339- strides), fix vertical filter kernel callback, fix threaded
340- gather buffer priming (and assert).
341- (thanks adipose, TainZerL, and Harrison Green)
342- 2.15 (2025-07-17) fixed an assert in debug mode when using floats with input
343- callbacks, work around GCC warning when adding to null ptr
344- (thanks Johannes Spohr and Pyry Kovanen).
345- 2.14 (2025-05-09) fixed a bug using downsampling gather horizontal first, and
346- scatter with vertical first.
347- 2.13 (2025-02-27) fixed a bug when using input callbacks, turned off simd for
348- tiny-c, fixed some variables that should have been static,
349- fixes a bug when calculating temp memory with resizes that
350- exceed 2GB of temp memory (very large resizes).
351- 2.12 (2024-10-18) fix incorrect use of user_data with STBIR_FREE
352- 2.11 (2024-09-08) fix harmless asan warnings in 2-channel and 3-channel mode
353- with AVX-2, fix some weird scaling edge conditions with
354- point sample mode.
355- 2.10 (2024-07-27) fix the defines GCC and mingw for loop unroll control,
356- fix MSVC 32-bit arm half float routines.
357- 2.09 (2024-06-19) fix the defines for 32-bit ARM GCC builds (was selecting
358- hardware half floats).
359- 2.08 (2024-06-10) fix for RGB->BGR three channel flips and add SIMD (thanks
360- to Ryan Salsbury), fix for sub-rect resizes, use the
361- pragmas to control unrolling when they are available.
362- 2.07 (2024-05-24) fix for slow final split during threaded conversions of very
363- wide scanlines when downsampling (caused by extra input
364- converting), fix for wide scanline resamples with many
365- splits (int overflow), fix GCC warning.
366- 2.06 (2024-02-10) fix for identical width/height 3x or more down-scaling
367- undersampling a single row on rare resize ratios (about 1%).
368- 2.05 (2024-02-07) fix for 2 pixel to 1 pixel resizes with wrap (thanks Aras),
369- fix for output callback (thanks Julien Koenen).
370- 2.04 (2023-11-17) fix for rare AVX bug, shadowed symbol (thanks Nikola Smiljanic).
371- 2.03 (2023-11-01) ASAN and TSAN warnings fixed, minor tweaks.
372- 2.00 (2023-10-10) mostly new source: new api, optimizations, simd, vertical-first, etc
373- 2x-5x faster without simd, 4x-12x faster with simd,
374- in some cases, 20x to 40x faster esp resizing large to very small.
375- 0.96 (2019-03-04) fixed warnings
376- 0.95 (2017-07-23) fixed warnings
377- 0.94 (2017-03-18) fixed warnings
378- 0.93 (2017-03-03) fixed bug with certain combinations of heights
379- 0.92 (2017-01-02) fix integer overflow on large (>2GB) images
380- 0.91 (2016-04-02) fix warnings; fix handling of subpixel regions
381- 0.90 (2014-09-17) first released version
382-
383- LICENSE
384- See end of file for license information.
385-*/
386-
387-#if !defined(STB_IMAGE_RESIZE_DO_HORIZONTALS) && !defined(STB_IMAGE_RESIZE_DO_VERTICALS) && !defined(STB_IMAGE_RESIZE_DO_CODERS) // for internal re-includes
388-
389-#ifndef STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
390-#define STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
391-
392-#include <stddef.h>
393-#ifdef _MSC_VER
394-typedef unsigned char stbir_uint8;
395-typedef unsigned short stbir_uint16;
396-typedef unsigned int stbir_uint32;
397-typedef unsigned __int64 stbir_uint64;
398-#else
399-#include <stdint.h>
400-typedef uint8_t stbir_uint8;
401-typedef uint16_t stbir_uint16;
402-typedef uint32_t stbir_uint32;
403-typedef uint64_t stbir_uint64;
404-#endif
405-
406-#ifndef STBIRDEF
407-#ifdef STB_IMAGE_RESIZE_STATIC
408-#define STBIRDEF static
409-#else
410-#ifdef __cplusplus
411-#define STBIRDEF extern "C"
412-#else
413-#define STBIRDEF extern
414-#endif
415-#endif
416-#endif
417-
418-//////////////////////////////////////////////////////////////////////////////
419-//// start "header file" ///////////////////////////////////////////////////
420-//
421-// Easy-to-use API:
422-//
423-// * stride is the offset between successive rows of image data
424-// in memory, in bytes. specify 0 for packed continuously in memory
425-// * colorspace is linear or sRGB as specified by function name
426-// * Uses the default filters
427-// * Uses edge mode clamped
428-// * returned result is 1 for success or 0 in case of an error.
429-
430-
431-// stbir_pixel_layout specifies:
432-// number of channels
433-// order of channels
434-// whether color is premultiplied by alpha
435-// for back compatibility, you can cast the old channel count to an stbir_pixel_layout
436-typedef enum
437-{
438- STBIR_1CHANNEL = 1,
439- STBIR_2CHANNEL = 2,
440- STBIR_RGB = 3, // 3-chan, with order specified (for channel flipping)
441- STBIR_BGR = 0, // 3-chan, with order specified (for channel flipping)
442- STBIR_4CHANNEL = 5,
443-
444- STBIR_RGBA = 4, // alpha formats, where alpha is NOT premultiplied into color channels
445- STBIR_BGRA = 6,
446- STBIR_ARGB = 7,
447- STBIR_ABGR = 8,
448- STBIR_RA = 9,
449- STBIR_AR = 10,
450-
451- STBIR_RGBA_PM = 11, // alpha formats, where alpha is premultiplied into color channels
452- STBIR_BGRA_PM = 12,
453- STBIR_ARGB_PM = 13,
454- STBIR_ABGR_PM = 14,
455- STBIR_RA_PM = 15,
456- STBIR_AR_PM = 16,
457-
458- STBIR_RGBA_NO_AW = 11, // alpha formats, where NO alpha weighting is applied at all!
459- STBIR_BGRA_NO_AW = 12, // these are just synonyms for the _PM flags (which also do
460- STBIR_ARGB_NO_AW = 13, // no alpha weighting). These names just make it more clear
461- STBIR_ABGR_NO_AW = 14, // for some folks).
462- STBIR_RA_NO_AW = 15,
463- STBIR_AR_NO_AW = 16,
464-
465-} stbir_pixel_layout;
466-
467-//===============================================================
468-// Simple-complexity API
469-//
470-// If output_pixels is NULL (0), then we will allocate the buffer and return it to you.
471-//--------------------------------
472-
473-STBIRDEF unsigned char * stbir_resize_uint8_srgb( const unsigned char *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
474- unsigned char *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
475- stbir_pixel_layout pixel_type );
476-
477-STBIRDEF unsigned char * stbir_resize_uint8_linear( const unsigned char *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
478- unsigned char *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
479- stbir_pixel_layout pixel_type );
480-
481-STBIRDEF float * stbir_resize_float_linear( const float *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
482- float *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
483- stbir_pixel_layout pixel_type );
484-//===============================================================
485-
486-//===============================================================
487-// Medium-complexity API
488-//
489-// This extends the easy-to-use API as follows:
490-//
491-// * Can specify the datatype - U8, U8_SRGB, U16, FLOAT, HALF_FLOAT
492-// * Edge wrap can selected explicitly
493-// * Filter can be selected explicitly
494-//--------------------------------
495-
496-typedef enum
497-{
498- STBIR_EDGE_CLAMP = 0,
499- STBIR_EDGE_REFLECT = 1,
500- STBIR_EDGE_WRAP = 2, // this edge mode is slower and uses more memory
501- STBIR_EDGE_ZERO = 3,
502-} stbir_edge;
503-
504-typedef enum
505-{
506- STBIR_FILTER_DEFAULT = 0, // use same filter type that easy-to-use API chooses
507- STBIR_FILTER_BOX = 1, // A trapezoid w/1-pixel wide ramps, same result as box for integer scale ratios
508- STBIR_FILTER_TRIANGLE = 2, // On upsampling, produces same results as bilinear texture filtering
509- STBIR_FILTER_CUBICBSPLINE = 3, // The cubic b-spline (aka Mitchell-Netrevalli with B=1,C=0), gaussian-esque
510- STBIR_FILTER_CATMULLROM = 4, // An interpolating cubic spline
511- STBIR_FILTER_MITCHELL = 5, // Mitchell-Netrevalli filter with B=1/3, C=1/3
512- STBIR_FILTER_POINT_SAMPLE = 6, // Simple point sampling
513- STBIR_FILTER_OTHER = 7, // User callback specified
514-} stbir_filter;
515-
516-typedef enum
517-{
518- STBIR_TYPE_UINT8 = 0,
519- STBIR_TYPE_UINT8_SRGB = 1,
520- STBIR_TYPE_UINT8_SRGB_ALPHA = 2, // alpha channel, when present, should also be SRGB (this is very unusual)
521- STBIR_TYPE_UINT16 = 3,
522- STBIR_TYPE_FLOAT = 4,
523- STBIR_TYPE_HALF_FLOAT = 5
524-} stbir_datatype;
525-
526-// medium api
527-STBIRDEF void * stbir_resize( const void *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
528- void *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
529- stbir_pixel_layout pixel_layout, stbir_datatype data_type,
530- stbir_edge edge, stbir_filter filter );
531-//===============================================================
532-
533-
534-
535-//===============================================================
536-// Extended-complexity API
537-//
538-// This API exposes all resize functionality.
539-//
540-// * Separate filter types for each axis
541-// * Separate edge modes for each axis
542-// * Separate input and output data types
543-// * Can specify regions with subpixel correctness
544-// * Can specify alpha flags
545-// * Can specify a memory callback
546-// * Can specify a callback data type for pixel input and output
547-// * Can be threaded for a single resize
548-// * Can be used to resize many frames without recalculating the sampler info
549-//
550-// Use this API as follows:
551-// 1) Call the stbir_resize_init function on a local STBIR_RESIZE structure
552-// 2) Call any of the stbir_set functions
553-// 3) Optionally call stbir_build_samplers() if you are going to resample multiple times
554-// with the same input and output dimensions (like resizing video frames)
555-// 4) Resample by calling stbir_resize_extended().
556-// 5) Call stbir_free_samplers() if you called stbir_build_samplers()
557-//--------------------------------
558-
559-
560-// Types:
561-
562-// INPUT CALLBACK: this callback is used for input scanlines
563-typedef void const * stbir_input_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context );
564-
565-// OUTPUT CALLBACK: this callback is used for output scanlines
566-typedef void stbir_output_callback( void const * output_ptr, int num_pixels, int y, void * context );
567-
568-// callbacks for user installed filters
569-typedef float stbir__kernel_callback( float x, float scale, void * user_data ); // centered at zero
570-typedef float stbir__support_callback( float scale, void * user_data );
571-
572-// internal structure with precomputed scaling
573-typedef struct stbir__info stbir__info;
574-
575-typedef struct STBIR_RESIZE // use the stbir_resize_init and stbir_override functions to set these values for future compatibility
576-{
577- void * user_data;
578- void const * input_pixels;
579- int input_w, input_h;
580- double input_s0, input_t0, input_s1, input_t1;
581- stbir_input_callback * input_cb;
582- void * output_pixels;
583- int output_w, output_h;
584- int output_subx, output_suby, output_subw, output_subh;
585- stbir_output_callback * output_cb;
586- int input_stride_in_bytes;
587- int output_stride_in_bytes;
588- int splits;
589- int fast_alpha;
590- int needs_rebuild;
591- int called_alloc;
592- stbir_pixel_layout input_pixel_layout_public;
593- stbir_pixel_layout output_pixel_layout_public;
594- stbir_datatype input_data_type;
595- stbir_datatype output_data_type;
596- stbir_filter horizontal_filter, vertical_filter;
597- stbir_edge horizontal_edge, vertical_edge;
598- stbir__kernel_callback * horizontal_filter_kernel; stbir__support_callback * horizontal_filter_support;
599- stbir__kernel_callback * vertical_filter_kernel; stbir__support_callback * vertical_filter_support;
600- stbir__info * samplers;
601-} STBIR_RESIZE;
602-
603-// extended complexity api
604-
605-
606-// First off, you must ALWAYS call stbir_resize_init on your resize structure before any of the other calls!
607-STBIRDEF void stbir_resize_init( STBIR_RESIZE * resize,
608- const void *input_pixels, int input_w, int input_h, int input_stride_in_bytes, // stride can be zero
609- void *output_pixels, int output_w, int output_h, int output_stride_in_bytes, // stride can be zero
610- stbir_pixel_layout pixel_layout, stbir_datatype data_type );
611-
612-//===============================================================
613-// You can update these parameters any time after resize_init and there is no cost
614-//--------------------------------
615-
616-STBIRDEF void stbir_set_datatypes( STBIR_RESIZE * resize, stbir_datatype input_type, stbir_datatype output_type );
617-STBIRDEF void stbir_set_pixel_callbacks( STBIR_RESIZE * resize, stbir_input_callback * input_cb, stbir_output_callback * output_cb ); // no callbacks by default
618-STBIRDEF void stbir_set_user_data( STBIR_RESIZE * resize, void * user_data ); // pass back STBIR_RESIZE* by default
619-STBIRDEF void stbir_set_buffer_ptrs( STBIR_RESIZE * resize, const void * input_pixels, int input_stride_in_bytes, void * output_pixels, int output_stride_in_bytes );
620-
621-//===============================================================
622-
623-
624-//===============================================================
625-// If you call any of these functions, you will trigger a sampler rebuild!
626-//--------------------------------
627-
628-STBIRDEF int stbir_set_pixel_layouts( STBIR_RESIZE * resize, stbir_pixel_layout input_pixel_layout, stbir_pixel_layout output_pixel_layout ); // sets new buffer layouts
629-STBIRDEF int stbir_set_edgemodes( STBIR_RESIZE * resize, stbir_edge horizontal_edge, stbir_edge vertical_edge ); // CLAMP by default
630-
631-STBIRDEF int stbir_set_filters( STBIR_RESIZE * resize, stbir_filter horizontal_filter, stbir_filter vertical_filter ); // STBIR_DEFAULT_FILTER_UPSAMPLE/DOWNSAMPLE by default
632-STBIRDEF int stbir_set_filter_callbacks( STBIR_RESIZE * resize, stbir__kernel_callback * horizontal_filter, stbir__support_callback * horizontal_support, stbir__kernel_callback * vertical_filter, stbir__support_callback * vertical_support );
633-
634-STBIRDEF int stbir_set_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ); // sets both sub-regions (full regions by default)
635-STBIRDEF int stbir_set_input_subrect( STBIR_RESIZE * resize, double s0, double t0, double s1, double t1 ); // sets input sub-region (full region by default)
636-STBIRDEF int stbir_set_output_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ); // sets output sub-region (full region by default)
637-
638-// when inputting AND outputting non-premultiplied alpha pixels, we use a slower but higher quality technique
639-// that fills the zero alpha pixel's RGB values with something plausible. If you don't care about areas of
640-// zero alpha, you can call this function to get about a 25% speed improvement for STBIR_RGBA to STBIR_RGBA
641-// types of resizes.
642-STBIRDEF int stbir_set_non_pm_alpha_speed_over_quality( STBIR_RESIZE * resize, int non_pma_alpha_speed_over_quality );
643-//===============================================================
644-
645-
646-//===============================================================
647-// You can call build_samplers to prebuild all the internal data we need to resample.
648-// Then, if you call resize_extended many times with the same resize, you only pay the
649-// cost once.
650-// If you do call build_samplers, you MUST call free_samplers eventually.
651-//--------------------------------
652-
653-// This builds the samplers and does one allocation
654-STBIRDEF int stbir_build_samplers( STBIR_RESIZE * resize );
655-
656-// You MUST call this, if you call stbir_build_samplers or stbir_build_samplers_with_splits
657-STBIRDEF void stbir_free_samplers( STBIR_RESIZE * resize );
658-//===============================================================
659-
660-
661-// And this is the main function to perform the resize synchronously on one thread.
662-STBIRDEF int stbir_resize_extended( STBIR_RESIZE * resize );
663-
664-
665-//===============================================================
666-// Use these functions for multithreading.
667-// 1) You call stbir_build_samplers_with_splits first on the main thread
668-// 2) Then stbir_resize_with_split on each thread
669-// 3) stbir_free_samplers when done on the main thread
670-//--------------------------------
671-
672-// This will build samplers for threading.
673-// You can pass in the number of threads you'd like to use (try_splits).
674-// It returns the number of splits (threads) that you can call it with.
675-/// It might be less if the image resize can't be split up that many ways.
676-
677-STBIRDEF int stbir_build_samplers_with_splits( STBIR_RESIZE * resize, int try_splits );
678-
679-// This function does a split of the resizing (you call this fuction for each
680-// split, on multiple threads). A split is a piece of the output resize pixel space.
681-
682-// Note that you MUST call stbir_build_samplers_with_splits before stbir_resize_extended_split!
683-
684-// Usually, you will always call stbir_resize_split with split_start as the thread_index
685-// and "1" for the split_count.
686-// But, if you have a weird situation where you MIGHT want 8 threads, but sometimes
687-// only 4 threads, you can use 0,2,4,6 for the split_start's and use "2" for the
688-// split_count each time to turn in into a 4 thread resize. (This is unusual).
689-
690-STBIRDEF int stbir_resize_extended_split( STBIR_RESIZE * resize, int split_start, int split_count );
691-//===============================================================
692-
693-
694-//===============================================================
695-// Pixel Callbacks info:
696-//--------------------------------
697-
698-// The input callback is super flexible - it calls you with the input address
699-// (based on the stride and base pointer), it gives you an optional_output
700-// pointer that you can fill, or you can just return your own pointer into
701-// your own data.
702-//
703-// You can also do conversion from non-supported data types if necessary - in
704-// this case, you ignore the input_ptr and just use the x and y parameters to
705-// calculate your own input_ptr based on the size of each non-supported pixel.
706-// (Something like the third example below.)
707-//
708-// You can also install just an input or just an output callback by setting the
709-// callback that you don't want to zero.
710-//
711-// First example, progress: (getting a callback that you can monitor the progress):
712-// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
713-// {
714-// percentage_done = y / input_height;
715-// return input_ptr; // use buffer from call
716-// }
717-//
718-// Next example, copying: (copy from some other buffer or stream):
719-// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
720-// {
721-// CopyOrStreamData( optional_output, other_data_src, num_pixels * pixel_width_in_bytes );
722-// return optional_output; // return the optional buffer that we filled
723-// }
724-//
725-// Third example, input another buffer without copying: (zero-copy from other buffer):
726-// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
727-// {
728-// void * pixels = ( (char*) other_image_base ) + ( y * other_image_stride ) + ( x * other_pixel_width_in_bytes );
729-// return pixels; // return pointer to your data without copying
730-// }
731-//
732-//
733-// The output callback is considerably simpler - it just calls you so that you can dump
734-// out each scanline. You could even directly copy out to disk if you have a simple format
735-// like TGA or BMP. You can also convert to other output types here if you want.
736-//
737-// Simple example:
738-// void const * my_output( void * output_ptr, int num_pixels, int y, void * context )
739-// {
740-// percentage_done = y / output_height;
741-// fwrite( output_ptr, pixel_width_in_bytes, num_pixels, output_file );
742-// }
743-//===============================================================
744-
745-
746-
747-
748-//===============================================================
749-// optional built-in profiling API
750-//--------------------------------
751-
752-#ifdef STBIR_PROFILE
753-
754-typedef struct STBIR_PROFILE_INFO
755-{
756- stbir_uint64 total_clocks;
757-
758- // how many clocks spent (of total_clocks) in the various resize routines, along with a string description
759- // there are "resize_count" number of zones
760- stbir_uint64 clocks[ 8 ];
761- char const ** descriptions;
762-
763- // count of clocks and descriptions
764- stbir_uint32 count;
765-} STBIR_PROFILE_INFO;
766-
767-// use after calling stbir_resize_extended (or stbir_build_samplers or stbir_build_samplers_with_splits)
768-STBIRDEF void stbir_resize_build_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize );
769-
770-// use after calling stbir_resize_extended
771-STBIRDEF void stbir_resize_extended_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize );
772-
773-// use after calling stbir_resize_extended_split
774-STBIRDEF void stbir_resize_split_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize, int split_start, int split_num );
775-
776-//===============================================================
777-
778-#endif
779-
780-
781-//// end header file /////////////////////////////////////////////////////
782-#endif // STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
783-
784-#if defined(STB_IMAGE_RESIZE_IMPLEMENTATION) || defined(STB_IMAGE_RESIZE2_IMPLEMENTATION)
785-
786-#ifndef STBIR_ASSERT
787-#include <assert.h>
788-#define STBIR_ASSERT(x) assert(x)
789-#endif
790-
791-#ifndef STBIR_MALLOC
792-#include <stdlib.h>
793-#define STBIR_MALLOC(size,user_data) ((void)(user_data), malloc(size))
794-#define STBIR_FREE(ptr,user_data) ((void)(user_data), free(ptr))
795-// (we used the comma operator to evaluate user_data, to avoid "unused parameter" warnings)
796-#endif
797-
798-#ifdef _MSC_VER
799-
800-#define stbir__inline __forceinline
801-
802-#else
803-
804-#define stbir__inline __inline__
805-
806-// Clang address sanitizer
807-#if defined(__has_feature)
808- #if __has_feature(address_sanitizer) || __has_feature(memory_sanitizer)
809- #ifndef STBIR__SEPARATE_ALLOCATIONS
810- #define STBIR__SEPARATE_ALLOCATIONS
811- #endif
812- #endif
813-#endif
814-
815-#endif
816-
817-// GCC and MSVC
818-#if defined(__SANITIZE_ADDRESS__)
819- #ifndef STBIR__SEPARATE_ALLOCATIONS
820- #define STBIR__SEPARATE_ALLOCATIONS
821- #endif
822-#endif
823-
824-// Always turn off automatic FMA use - use STBIR_USE_FMA if you want.
825-// Otherwise, this is a determinism disaster.
826-#ifndef STBIR_DONT_CHANGE_FP_CONTRACT // override in case you don't want this behavior
827-#if defined(_MSC_VER) && !defined(__clang__)
828-#if _MSC_VER > 1200
829-#pragma fp_contract(off)
830-#endif
831-#elif defined(__GNUC__) && !defined(__clang__)
832-#pragma GCC optimize("fp-contract=off")
833-#else
834-#pragma STDC FP_CONTRACT OFF
835-#endif
836-#endif
837-
838-#ifdef _MSC_VER
839-#define STBIR__UNUSED(v) (void)(v)
840-#else
841-#define STBIR__UNUSED(v) (void)sizeof(v)
842-#endif
843-
844-#define STBIR__ARRAY_SIZE(a) (sizeof((a))/sizeof((a)[0]))
845-
846-
847-#ifndef STBIR_DEFAULT_FILTER_UPSAMPLE
848-#define STBIR_DEFAULT_FILTER_UPSAMPLE STBIR_FILTER_CATMULLROM
849-#endif
850-
851-#ifndef STBIR_DEFAULT_FILTER_DOWNSAMPLE
852-#define STBIR_DEFAULT_FILTER_DOWNSAMPLE STBIR_FILTER_MITCHELL
853-#endif
854-
855-
856-#ifndef STBIR__HEADER_FILENAME
857-#define STBIR__HEADER_FILENAME "stb_image_resize2.h"
858-#endif
859-
860-// the internal pixel layout enums are in a different order, so we can easily do range comparisons of types
861-// the public pixel layout is ordered in a way that if you cast num_channels (1-4) to the enum, you get something sensible
862-typedef enum
863-{
864- STBIRI_1CHANNEL = 0,
865- STBIRI_2CHANNEL = 1,
866- STBIRI_RGB = 2,
867- STBIRI_BGR = 3,
868- STBIRI_4CHANNEL = 4,
869-
870- STBIRI_RGBA = 5,
871- STBIRI_BGRA = 6,
872- STBIRI_ARGB = 7,
873- STBIRI_ABGR = 8,
874- STBIRI_RA = 9,
875- STBIRI_AR = 10,
876-
877- STBIRI_RGBA_PM = 11,
878- STBIRI_BGRA_PM = 12,
879- STBIRI_ARGB_PM = 13,
880- STBIRI_ABGR_PM = 14,
881- STBIRI_RA_PM = 15,
882- STBIRI_AR_PM = 16,
883-} stbir_internal_pixel_layout;
884-
885-// define the public pixel layouts to not compile inside the implementation (to avoid accidental use)
886-#define STBIR_BGR bad_dont_use_in_implementation
887-#define STBIR_1CHANNEL STBIR_BGR
888-#define STBIR_2CHANNEL STBIR_BGR
889-#define STBIR_RGB STBIR_BGR
890-#define STBIR_RGBA STBIR_BGR
891-#define STBIR_4CHANNEL STBIR_BGR
892-#define STBIR_BGRA STBIR_BGR
893-#define STBIR_ARGB STBIR_BGR
894-#define STBIR_ABGR STBIR_BGR
895-#define STBIR_RA STBIR_BGR
896-#define STBIR_AR STBIR_BGR
897-#define STBIR_RGBA_PM STBIR_BGR
898-#define STBIR_BGRA_PM STBIR_BGR
899-#define STBIR_ARGB_PM STBIR_BGR
900-#define STBIR_ABGR_PM STBIR_BGR
901-#define STBIR_RA_PM STBIR_BGR
902-#define STBIR_AR_PM STBIR_BGR
903-
904-// must match stbir_datatype
905-static unsigned char stbir__type_size[] = {
906- 1,1,1,2,4,2 // STBIR_TYPE_UINT8,STBIR_TYPE_UINT8_SRGB,STBIR_TYPE_UINT8_SRGB_ALPHA,STBIR_TYPE_UINT16,STBIR_TYPE_FLOAT,STBIR_TYPE_HALF_FLOAT
907-};
908-
909-// When gathering, the contributors are which source pixels contribute.
910-// When scattering, the contributors are which destination pixels are contributed to.
911-typedef struct
912-{
913- int n0; // First contributing pixel
914- int n1; // Last contributing pixel
915-} stbir__contributors;
916-
917-typedef struct
918-{
919- int lowest; // First sample index for whole filter
920- int highest; // Last sample index for whole filter
921- int widest; // widest single set of samples for an output
922-} stbir__filter_extent_info;
923-
924-typedef struct
925-{
926- int n0; // First pixel of decode buffer to write to
927- int n1; // Last pixel of decode that will be written to
928- int pixel_offset_for_input; // Pixel offset into input_scanline
929-} stbir__span;
930-
931-typedef struct stbir__scale_info
932-{
933- int input_full_size;
934- int output_sub_size;
935- float scale;
936- float inv_scale;
937- float pixel_shift; // starting shift in output pixel space (in pixels)
938- int scale_is_rational;
939- stbir_uint32 scale_numerator, scale_denominator;
940-} stbir__scale_info;
941-
942-typedef struct
943-{
944- stbir__contributors * contributors;
945- float* coefficients;
946- stbir__contributors * gather_prescatter_contributors;
947- float * gather_prescatter_coefficients;
948- stbir__scale_info scale_info;
949- float support;
950- stbir_filter filter_enum;
951- stbir__kernel_callback * filter_kernel;
952- stbir__support_callback * filter_support;
953- stbir_edge edge;
954- int coefficient_width;
955- int filter_pixel_width;
956- int filter_pixel_margin;
957- int num_contributors;
958- int contributors_size;
959- int coefficients_size;
960- stbir__filter_extent_info extent_info;
961- int is_gather; // 0 = scatter, 1 = gather with scale >= 1, 2 = gather with scale < 1
962- int gather_prescatter_num_contributors;
963- int gather_prescatter_coefficient_width;
964- int gather_prescatter_contributors_size;
965- int gather_prescatter_coefficients_size;
966-} stbir__sampler;
967-
968-typedef struct
969-{
970- stbir__contributors conservative;
971- int edge_sizes[2]; // this can be less than filter_pixel_margin, if the filter and scaling falls off
972- stbir__span spans[2]; // can be two spans, if doing input subrect with clamp mode WRAP
973-} stbir__extents;
974-
975-typedef struct
976-{
977-#ifdef STBIR_PROFILE
978- union
979- {
980- struct { stbir_uint64 total, looping, vertical, horizontal, decode, encode, alpha, unalpha; } named;
981- stbir_uint64 array[8];
982- } profile;
983- stbir_uint64 * current_zone_excluded_ptr;
984-#endif
985- float* decode_buffer;
986-
987- int ring_buffer_first_scanline;
988- int ring_buffer_last_scanline;
989- int ring_buffer_begin_index; // first_scanline is at this index in the ring buffer
990- int start_output_y, end_output_y;
991- int start_input_y, end_input_y; // used in scatter only
992-
993- #ifdef STBIR__SEPARATE_ALLOCATIONS
994- float** ring_buffers; // one pointer for each ring buffer
995- #else
996- float* ring_buffer; // one big buffer that we index into
997- #endif
998-
999- float* vertical_buffer;
1000-
1001- char no_cache_straddle[64];
1002-} stbir__per_split_info;
1003-
1004-typedef float * stbir__decode_pixels_func( float * decode, int width_times_channels, void const * input );
1005-typedef void stbir__alpha_weight_func( float * decode_buffer, int width_times_channels );
1006-typedef void stbir__horizontal_gather_channels_func( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer,
1007- stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width );
1008-typedef void stbir__alpha_unweight_func(float * encode_buffer, int width_times_channels );
1009-typedef void stbir__encode_pixels_func( void * output, int width_times_channels, float const * encode );
1010-
1011-struct stbir__info
1012-{
1013-#ifdef STBIR_PROFILE
1014- union
1015- {
1016- struct { stbir_uint64 total, build, alloc, horizontal, vertical, cleanup, pivot; } named;
1017- stbir_uint64 array[7];
1018- } profile;
1019- stbir_uint64 * current_zone_excluded_ptr;
1020-#endif
1021- stbir__sampler horizontal;
1022- stbir__sampler vertical;
1023-
1024- void const * input_data;
1025- void * output_data;
1026-
1027- int input_stride_bytes;
1028- int output_stride_bytes;
1029- int ring_buffer_length_bytes; // The length of an individual entry in the ring buffer. The total number of ring buffers is stbir__get_filter_pixel_width(filter)
1030- int ring_buffer_num_entries; // Total number of entries in the ring buffer.
1031-
1032- stbir_datatype input_type;
1033- stbir_datatype output_type;
1034-
1035- stbir_input_callback * in_pixels_cb;
1036- void * user_data;
1037- stbir_output_callback * out_pixels_cb;
1038-
1039- stbir__extents scanline_extents;
1040-
1041- void * alloced_mem;
1042- stbir__per_split_info * split_info; // by default 1, but there will be N of these allocated based on the thread init you did
1043-
1044- stbir__decode_pixels_func * decode_pixels;
1045- stbir__alpha_weight_func * alpha_weight;
1046- stbir__horizontal_gather_channels_func * horizontal_gather_channels;
1047- stbir__alpha_unweight_func * alpha_unweight;
1048- stbir__encode_pixels_func * encode_pixels;
1049-
1050- int alloc_ring_buffer_num_entries; // Number of entries in the ring buffer that will be allocated
1051- int splits; // count of splits
1052-
1053- stbir_internal_pixel_layout input_pixel_layout_internal;
1054- stbir_internal_pixel_layout output_pixel_layout_internal;
1055-
1056- int input_color_and_type;
1057- int offset_x, offset_y; // offset within output_data
1058- int vertical_first;
1059- int channels;
1060- int effective_channels; // same as channels, except on RGBA/ARGB (7), or XA/AX (3)
1061- size_t alloced_total;
1062-};
1063-
1064-
1065-#define stbir__max_uint8_as_float 255.0f
1066-#define stbir__max_uint16_as_float 65535.0f
1067-#define stbir__max_uint8_as_float_inverted 3.9215689e-03f // (1.0f/255.0f)
1068-#define stbir__max_uint16_as_float_inverted 1.5259022e-05f // (1.0f/65535.0f)
1069-#define stbir__small_float ((float)1 / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20))
1070-
1071-// min/max friendly
1072-#define STBIR_CLAMP(x, xmin, xmax) for(;;) { \
1073- if ( (x) < (xmin) ) (x) = (xmin); \
1074- if ( (x) > (xmax) ) (x) = (xmax); \
1075- break; \
1076-}
1077-
1078-static stbir__inline int stbir__min(int a, int b)
1079-{
1080- return a < b ? a : b;
1081-}
1082-
1083-static stbir__inline int stbir__max(int a, int b)
1084-{
1085- return a > b ? a : b;
1086-}
1087-
1088-static float stbir__srgb_uchar_to_linear_float[256] = {
1089- 0.000000f, 0.000304f, 0.000607f, 0.000911f, 0.001214f, 0.001518f, 0.001821f, 0.002125f, 0.002428f, 0.002732f, 0.003035f,
1090- 0.003347f, 0.003677f, 0.004025f, 0.004391f, 0.004777f, 0.005182f, 0.005605f, 0.006049f, 0.006512f, 0.006995f, 0.007499f,
1091- 0.008023f, 0.008568f, 0.009134f, 0.009721f, 0.010330f, 0.010960f, 0.011612f, 0.012286f, 0.012983f, 0.013702f, 0.014444f,
1092- 0.015209f, 0.015996f, 0.016807f, 0.017642f, 0.018500f, 0.019382f, 0.020289f, 0.021219f, 0.022174f, 0.023153f, 0.024158f,
1093- 0.025187f, 0.026241f, 0.027321f, 0.028426f, 0.029557f, 0.030713f, 0.031896f, 0.033105f, 0.034340f, 0.035601f, 0.036889f,
1094- 0.038204f, 0.039546f, 0.040915f, 0.042311f, 0.043735f, 0.045186f, 0.046665f, 0.048172f, 0.049707f, 0.051269f, 0.052861f,
1095- 0.054480f, 0.056128f, 0.057805f, 0.059511f, 0.061246f, 0.063010f, 0.064803f, 0.066626f, 0.068478f, 0.070360f, 0.072272f,
1096- 0.074214f, 0.076185f, 0.078187f, 0.080220f, 0.082283f, 0.084376f, 0.086500f, 0.088656f, 0.090842f, 0.093059f, 0.095307f,
1097- 0.097587f, 0.099899f, 0.102242f, 0.104616f, 0.107023f, 0.109462f, 0.111932f, 0.114435f, 0.116971f, 0.119538f, 0.122139f,
1098- 0.124772f, 0.127438f, 0.130136f, 0.132868f, 0.135633f, 0.138432f, 0.141263f, 0.144128f, 0.147027f, 0.149960f, 0.152926f,
1099- 0.155926f, 0.158961f, 0.162029f, 0.165132f, 0.168269f, 0.171441f, 0.174647f, 0.177888f, 0.181164f, 0.184475f, 0.187821f,
1100- 0.191202f, 0.194618f, 0.198069f, 0.201556f, 0.205079f, 0.208637f, 0.212231f, 0.215861f, 0.219526f, 0.223228f, 0.226966f,
1101- 0.230740f, 0.234551f, 0.238398f, 0.242281f, 0.246201f, 0.250158f, 0.254152f, 0.258183f, 0.262251f, 0.266356f, 0.270498f,
1102- 0.274677f, 0.278894f, 0.283149f, 0.287441f, 0.291771f, 0.296138f, 0.300544f, 0.304987f, 0.309469f, 0.313989f, 0.318547f,
1103- 0.323143f, 0.327778f, 0.332452f, 0.337164f, 0.341914f, 0.346704f, 0.351533f, 0.356400f, 0.361307f, 0.366253f, 0.371238f,
1104- 0.376262f, 0.381326f, 0.386430f, 0.391573f, 0.396755f, 0.401978f, 0.407240f, 0.412543f, 0.417885f, 0.423268f, 0.428691f,
1105- 0.434154f, 0.439657f, 0.445201f, 0.450786f, 0.456411f, 0.462077f, 0.467784f, 0.473532f, 0.479320f, 0.485150f, 0.491021f,
1106- 0.496933f, 0.502887f, 0.508881f, 0.514918f, 0.520996f, 0.527115f, 0.533276f, 0.539480f, 0.545725f, 0.552011f, 0.558340f,
1107- 0.564712f, 0.571125f, 0.577581f, 0.584078f, 0.590619f, 0.597202f, 0.603827f, 0.610496f, 0.617207f, 0.623960f, 0.630757f,
1108- 0.637597f, 0.644480f, 0.651406f, 0.658375f, 0.665387f, 0.672443f, 0.679543f, 0.686685f, 0.693872f, 0.701102f, 0.708376f,
1109- 0.715694f, 0.723055f, 0.730461f, 0.737911f, 0.745404f, 0.752942f, 0.760525f, 0.768151f, 0.775822f, 0.783538f, 0.791298f,
1110- 0.799103f, 0.806952f, 0.814847f, 0.822786f, 0.830770f, 0.838799f, 0.846873f, 0.854993f, 0.863157f, 0.871367f, 0.879622f,
1111- 0.887923f, 0.896269f, 0.904661f, 0.913099f, 0.921582f, 0.930111f, 0.938686f, 0.947307f, 0.955974f, 0.964686f, 0.973445f,
1112- 0.982251f, 0.991102f, 1.0f
1113-};
1114-
1115-typedef union
1116-{
1117- unsigned int u;
1118- float f;
1119-} stbir__FP32;
1120-
1121-// From https://gist.github.com/rygorous/2203834
1122-
1123-static const stbir_uint32 fp32_to_srgb8_tab4[104] = {
1124- 0x0073000d, 0x007a000d, 0x0080000d, 0x0087000d, 0x008d000d, 0x0094000d, 0x009a000d, 0x00a1000d,
1125- 0x00a7001a, 0x00b4001a, 0x00c1001a, 0x00ce001a, 0x00da001a, 0x00e7001a, 0x00f4001a, 0x0101001a,
1126- 0x010e0033, 0x01280033, 0x01410033, 0x015b0033, 0x01750033, 0x018f0033, 0x01a80033, 0x01c20033,
1127- 0x01dc0067, 0x020f0067, 0x02430067, 0x02760067, 0x02aa0067, 0x02dd0067, 0x03110067, 0x03440067,
1128- 0x037800ce, 0x03df00ce, 0x044600ce, 0x04ad00ce, 0x051400ce, 0x057b00c5, 0x05dd00bc, 0x063b00b5,
1129- 0x06970158, 0x07420142, 0x07e30130, 0x087b0120, 0x090b0112, 0x09940106, 0x0a1700fc, 0x0a9500f2,
1130- 0x0b0f01cb, 0x0bf401ae, 0x0ccb0195, 0x0d950180, 0x0e56016e, 0x0f0d015e, 0x0fbc0150, 0x10630143,
1131- 0x11070264, 0x1238023e, 0x1357021d, 0x14660201, 0x156601e9, 0x165a01d3, 0x174401c0, 0x182401af,
1132- 0x18fe0331, 0x1a9602fe, 0x1c1502d2, 0x1d7e02ad, 0x1ed4028d, 0x201a0270, 0x21520256, 0x227d0240,
1133- 0x239f0443, 0x25c003fe, 0x27bf03c4, 0x29a10392, 0x2b6a0367, 0x2d1d0341, 0x2ebe031f, 0x304d0300,
1134- 0x31d105b0, 0x34a80555, 0x37520507, 0x39d504c5, 0x3c37048b, 0x3e7c0458, 0x40a8042a, 0x42bd0401,
1135- 0x44c20798, 0x488e071e, 0x4c1c06b6, 0x4f76065d, 0x52a50610, 0x55ac05cc, 0x5892058f, 0x5b590559,
1136- 0x5e0c0a23, 0x631c0980, 0x67db08f6, 0x6c55087f, 0x70940818, 0x74a007bd, 0x787d076c, 0x7c330723,
1137-};
1138-
1139-static stbir__inline stbir_uint8 stbir__linear_to_srgb_uchar(float in)
1140-{
1141- static const stbir__FP32 almostone = { 0x3f7fffff }; // 1-eps
1142- static const stbir__FP32 minval = { (127-13) << 23 };
1143- stbir_uint32 tab,bias,scale,t;
1144- stbir__FP32 f;
1145-
1146- // Clamp to [2^(-13), 1-eps]; these two values map to 0 and 1, respectively.
1147- // The tests are carefully written so that NaNs map to 0, same as in the reference
1148- // implementation.
1149- if (!(in > minval.f)) // written this way to catch NaNs
1150- return 0;
1151- if (in > almostone.f)
1152- return 255;
1153-
1154- // Do the table lookup and unpack bias, scale
1155- f.f = in;
1156- tab = fp32_to_srgb8_tab4[(f.u - minval.u) >> 20];
1157- bias = (tab >> 16) << 9;
1158- scale = tab & 0xffff;
1159-
1160- // Grab next-highest mantissa bits and perform linear interpolation
1161- t = (f.u >> 12) & 0xff;
1162- return (unsigned char) ((bias + scale*t) >> 16);
1163-}
1164-
1165-#ifndef STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT
1166-#define STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT 32 // when downsampling and <= 32 scanlines of buffering, use gather. gather used down to 1/8th scaling for 25% win.
1167-#endif
1168-
1169-#ifndef STBIR_FORCE_MINIMUM_SCANLINES_FOR_SPLITS
1170-#define STBIR_FORCE_MINIMUM_SCANLINES_FOR_SPLITS 4 // when threading, what is the minimum number of scanlines for a split?
1171-#endif
1172-
1173-#define STBIR_INPUT_CALLBACK_PADDING 3
1174-
1175-#ifdef _M_IX86_FP
1176-#if ( _M_IX86_FP >= 1 )
1177-#ifndef STBIR_SSE
1178-#define STBIR_SSE
1179-#endif
1180-#endif
1181-#endif
1182-
1183-#ifdef __TINYC__
1184- // tiny c has no intrinsics yet - this can become a version check if they add them
1185- #define STBIR_NO_SIMD
1186-#endif
1187-
1188-#if defined(_x86_64) || defined( __x86_64__ ) || defined( _M_X64 ) || defined(__x86_64) || defined(_M_AMD64) || defined(__SSE2__) || defined(STBIR_SSE) || defined(STBIR_SSE2)
1189- #ifndef STBIR_SSE2
1190- #define STBIR_SSE2
1191- #endif
1192- #if defined(__AVX__) || defined(STBIR_AVX2)
1193- #ifndef STBIR_AVX
1194- #ifndef STBIR_NO_AVX
1195- #define STBIR_AVX
1196- #endif
1197- #endif
1198- #endif
1199- #if defined(__AVX2__) || defined(STBIR_AVX2)
1200- #ifndef STBIR_NO_AVX2
1201- #ifndef STBIR_AVX2
1202- #define STBIR_AVX2
1203- #endif
1204- #if defined( _MSC_VER ) && !defined(__clang__)
1205- #ifndef STBIR_FP16C // FP16C instructions are on all AVX2 cpus, so we can autoselect it here on microsoft - clang needs -mf16c
1206- #define STBIR_FP16C
1207- #endif
1208- #endif
1209- #endif
1210- #endif
1211- #ifdef __F16C__
1212- #ifndef STBIR_FP16C // turn on FP16C instructions if the define is set (for clang and gcc)
1213- #define STBIR_FP16C
1214- #endif
1215- #endif
1216-#endif
1217-
1218-#if defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ ) || ((__ARM_NEON_FP & 4) != 0) || defined(__ARM_NEON__)
1219-#ifndef STBIR_NEON
1220-#define STBIR_NEON
1221-#endif
1222-#endif
1223-
1224-#if defined(_M_ARM) || defined(__arm__)
1225-#ifdef STBIR_USE_FMA
1226-#undef STBIR_USE_FMA // no FMA for 32-bit arm on MSVC
1227-#endif
1228-#endif
1229-
1230-#if defined(__wasm__) && defined(__wasm_simd128__)
1231-#ifndef STBIR_WASM
1232-#define STBIR_WASM
1233-#endif
1234-#endif
1235-
1236-// restrict pointers for the output pointers, other loop and unroll control
1237-#if defined( _MSC_VER ) && !defined(__clang__)
1238- #define STBIR_STREAMOUT_PTR( star ) star __restrict
1239- #define STBIR_NO_UNROLL( ptr ) __assume(ptr) // this oddly keeps msvc from unrolling a loop
1240- #if _MSC_VER >= 1900
1241- #define STBIR_NO_UNROLL_LOOP_START __pragma(loop( no_vector ))
1242- #else
1243- #define STBIR_NO_UNROLL_LOOP_START
1244- #endif
1245-#elif defined( __clang__ )
1246- #define STBIR_STREAMOUT_PTR( star ) star __restrict__
1247- #define STBIR_NO_UNROLL( ptr ) __asm__ (""::"r"(ptr))
1248- #if ( __clang_major__ >= 4 ) || ( ( __clang_major__ >= 3 ) && ( __clang_minor__ >= 5 ) )
1249- #define STBIR_NO_UNROLL_LOOP_START _Pragma("clang loop unroll(disable)") _Pragma("clang loop vectorize(disable)")
1250- #else
1251- #define STBIR_NO_UNROLL_LOOP_START
1252- #endif
1253-#elif defined( __GNUC__ )
1254- #define STBIR_STREAMOUT_PTR( star ) star __restrict__
1255- #define STBIR_NO_UNROLL( ptr ) __asm__ (""::"r"(ptr))
1256- #if __GNUC__ >= 14
1257- #define STBIR_NO_UNROLL_LOOP_START _Pragma("GCC unroll 0") _Pragma("GCC novector")
1258- #else
1259- #define STBIR_NO_UNROLL_LOOP_START
1260- #endif
1261- #define STBIR_NO_UNROLL_LOOP_START_INF_FOR
1262-#else
1263- #define STBIR_STREAMOUT_PTR( star ) star
1264- #define STBIR_NO_UNROLL( ptr )
1265- #define STBIR_NO_UNROLL_LOOP_START
1266-#endif
1267-
1268-#ifndef STBIR_NO_UNROLL_LOOP_START_INF_FOR
1269-#define STBIR_NO_UNROLL_LOOP_START_INF_FOR STBIR_NO_UNROLL_LOOP_START
1270-#endif
1271-
1272-#ifdef STBIR_NO_SIMD // force simd off for whatever reason
1273-
1274-// force simd off overrides everything else, so clear it all
1275-
1276-#ifdef STBIR_SSE2
1277-#undef STBIR_SSE2
1278-#endif
1279-
1280-#ifdef STBIR_AVX
1281-#undef STBIR_AVX
1282-#endif
1283-
1284-#ifdef STBIR_NEON
1285-#undef STBIR_NEON
1286-#endif
1287-
1288-#ifdef STBIR_AVX2
1289-#undef STBIR_AVX2
1290-#endif
1291-
1292-#ifdef STBIR_FP16C
1293-#undef STBIR_FP16C
1294-#endif
1295-
1296-#ifdef STBIR_WASM
1297-#undef STBIR_WASM
1298-#endif
1299-
1300-#ifdef STBIR_SIMD
1301-#undef STBIR_SIMD
1302-#endif
1303-
1304-#else // STBIR_SIMD
1305-
1306-#ifdef STBIR_SSE2
1307- #include <emmintrin.h>
1308-
1309- #define stbir__simdf __m128
1310- #define stbir__simdi __m128i
1311-
1312- #define stbir_simdi_castf( reg ) _mm_castps_si128(reg)
1313- #define stbir_simdf_casti( reg ) _mm_castsi128_ps(reg)
1314-
1315- #define stbir__simdf_load( reg, ptr ) (reg) = _mm_loadu_ps( (float const*)(ptr) )
1316- #define stbir__simdi_load( reg, ptr ) (reg) = _mm_loadu_si128 ( (stbir__simdi const*)(ptr) )
1317- #define stbir__simdf_load1( out, ptr ) (out) = _mm_load_ss( (float const*)(ptr) ) // top values can be random (not denormal or nan for perf)
1318- #define stbir__simdi_load1( out, ptr ) (out) = _mm_castps_si128( _mm_load_ss( (float const*)(ptr) ))
1319- #define stbir__simdf_load1z( out, ptr ) (out) = _mm_load_ss( (float const*)(ptr) ) // top values must be zero
1320- #define stbir__simdf_frep4( fvar ) _mm_set_ps1( fvar )
1321- #define stbir__simdf_load1frep4( out, fvar ) (out) = _mm_set_ps1( fvar )
1322- #define stbir__simdf_load2( out, ptr ) (out) = _mm_castsi128_ps( _mm_loadl_epi64( (__m128i*)(ptr)) ) // top values can be random (not denormal or nan for perf)
1323- #define stbir__simdf_load2z( out, ptr ) (out) = _mm_castsi128_ps( _mm_loadl_epi64( (__m128i*)(ptr)) ) // top values must be zero
1324- #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = _mm_castpd_ps(_mm_loadh_pd( _mm_castps_pd(reg), (double*)(ptr) ))
1325-
1326- #define stbir__simdf_zeroP() _mm_setzero_ps()
1327- #define stbir__simdf_zero( reg ) (reg) = _mm_setzero_ps()
1328-
1329- #define stbir__simdf_store( ptr, reg ) _mm_storeu_ps( (float*)(ptr), reg )
1330- #define stbir__simdf_store1( ptr, reg ) _mm_store_ss( (float*)(ptr), reg )
1331- #define stbir__simdf_store2( ptr, reg ) _mm_storel_epi64( (__m128i*)(ptr), _mm_castps_si128(reg) )
1332- #define stbir__simdf_store2h( ptr, reg ) _mm_storeh_pd( (double*)(ptr), _mm_castps_pd(reg) )
1333-
1334- #define stbir__simdi_store( ptr, reg ) _mm_storeu_si128( (__m128i*)(ptr), reg )
1335- #define stbir__simdi_store1( ptr, reg ) _mm_store_ss( (float*)(ptr), _mm_castsi128_ps(reg) )
1336- #define stbir__simdi_store2( ptr, reg ) _mm_storel_epi64( (__m128i*)(ptr), (reg) )
1337-
1338- #define stbir__prefetch( ptr ) _mm_prefetch((char*)(ptr), _MM_HINT_T0 )
1339-
1340- #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
1341- { \
1342- stbir__simdi zero = _mm_setzero_si128(); \
1343- out2 = _mm_unpacklo_epi8( ireg, zero ); \
1344- out3 = _mm_unpackhi_epi8( ireg, zero ); \
1345- out0 = _mm_unpacklo_epi16( out2, zero ); \
1346- out1 = _mm_unpackhi_epi16( out2, zero ); \
1347- out2 = _mm_unpacklo_epi16( out3, zero ); \
1348- out3 = _mm_unpackhi_epi16( out3, zero ); \
1349- }
1350-
1351-#define stbir__simdi_expand_u8_to_1u32(out,ireg) \
1352- { \
1353- stbir__simdi zero = _mm_setzero_si128(); \
1354- out = _mm_unpacklo_epi8( ireg, zero ); \
1355- out = _mm_unpacklo_epi16( out, zero ); \
1356- }
1357-
1358- #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
1359- { \
1360- stbir__simdi zero = _mm_setzero_si128(); \
1361- out0 = _mm_unpacklo_epi16( ireg, zero ); \
1362- out1 = _mm_unpackhi_epi16( ireg, zero ); \
1363- }
1364-
1365- #define stbir__simdf_convert_float_to_i32( i, f ) (i) = _mm_cvttps_epi32(f)
1366- #define stbir__simdf_convert_float_to_int( f ) _mm_cvtt_ss2si(f)
1367- #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)_mm_cvtsi128_si32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(f,STBIR__CONSTF(STBIR_max_uint8_as_float)),_mm_setzero_ps()))))
1368- #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)_mm_cvtsi128_si32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(f,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps()))))
1369-
1370- #define stbir__simdi_to_int( i ) _mm_cvtsi128_si32(i)
1371- #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = _mm_cvtepi32_ps( ireg )
1372- #define stbir__simdf_add( out, reg0, reg1 ) (out) = _mm_add_ps( reg0, reg1 )
1373- #define stbir__simdf_mult( out, reg0, reg1 ) (out) = _mm_mul_ps( reg0, reg1 )
1374- #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = _mm_mul_ps( reg, _mm_loadu_ps( (float const*)(ptr) ) )
1375- #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = _mm_mul_ss( reg, _mm_load_ss( (float const*)(ptr) ) )
1376- #define stbir__simdf_add_mem( out, reg, ptr ) (out) = _mm_add_ps( reg, _mm_loadu_ps( (float const*)(ptr) ) )
1377- #define stbir__simdf_add1_mem( out, reg, ptr ) (out) = _mm_add_ss( reg, _mm_load_ss( (float const*)(ptr) ) )
1378-
1379- #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd
1380- #include <immintrin.h>
1381- #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = _mm_fmadd_ps( mul1, mul2, add )
1382- #define stbir__simdf_madd1( out, add, mul1, mul2 ) (out) = _mm_fmadd_ss( mul1, mul2, add )
1383- #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = _mm_fmadd_ps( mul, _mm_loadu_ps( (float const*)(ptr) ), add )
1384- #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = _mm_fmadd_ss( mul, _mm_load_ss( (float const*)(ptr) ), add )
1385- #else
1386- #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = _mm_add_ps( add, _mm_mul_ps( mul1, mul2 ) )
1387- #define stbir__simdf_madd1( out, add, mul1, mul2 ) (out) = _mm_add_ss( add, _mm_mul_ss( mul1, mul2 ) )
1388- #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = _mm_add_ps( add, _mm_mul_ps( mul, _mm_loadu_ps( (float const*)(ptr) ) ) )
1389- #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = _mm_add_ss( add, _mm_mul_ss( mul, _mm_load_ss( (float const*)(ptr) ) ) )
1390- #endif
1391-
1392- #define stbir__simdf_add1( out, reg0, reg1 ) (out) = _mm_add_ss( reg0, reg1 )
1393- #define stbir__simdf_mult1( out, reg0, reg1 ) (out) = _mm_mul_ss( reg0, reg1 )
1394-
1395- #define stbir__simdf_and( out, reg0, reg1 ) (out) = _mm_and_ps( reg0, reg1 )
1396- #define stbir__simdf_or( out, reg0, reg1 ) (out) = _mm_or_ps( reg0, reg1 )
1397-
1398- #define stbir__simdf_min( out, reg0, reg1 ) (out) = _mm_min_ps( reg0, reg1 )
1399- #define stbir__simdf_max( out, reg0, reg1 ) (out) = _mm_max_ps( reg0, reg1 )
1400- #define stbir__simdf_min1( out, reg0, reg1 ) (out) = _mm_min_ss( reg0, reg1 )
1401- #define stbir__simdf_max1( out, reg0, reg1 ) (out) = _mm_max_ss( reg0, reg1 )
1402-
1403- #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_shuffle_ps( reg1,reg0, (0<<0) + (1<<2) + (2<<4) + (3<<6) )), (3<<0) + (0<<2) + (1<<4) + (2<<6) ) )
1404- #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_shuffle_ps( reg1,reg0, (0<<0) + (1<<2) + (2<<4) + (3<<6) )), (2<<0) + (3<<2) + (0<<4) + (1<<6) ) )
1405-
1406- static const stbir__simdf STBIR_zeroones = { 0.0f,1.0f,0.0f,1.0f };
1407- static const stbir__simdf STBIR_onezeros = { 1.0f,0.0f,1.0f,0.0f };
1408- #define stbir__simdf_aaa1( out, alp, ones ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_movehl_ps( ones, alp ) ), (1<<0) + (1<<2) + (1<<4) + (2<<6) ) )
1409- #define stbir__simdf_1aaa( out, alp, ones ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_movelh_ps( ones, alp ) ), (0<<0) + (2<<2) + (2<<4) + (2<<6) ) )
1410- #define stbir__simdf_a1a1( out, alp, ones) (out) = _mm_or_ps( _mm_castsi128_ps( _mm_srli_epi64( _mm_castps_si128(alp), 32 ) ), STBIR_zeroones )
1411- #define stbir__simdf_1a1a( out, alp, ones) (out) = _mm_or_ps( _mm_castsi128_ps( _mm_slli_epi64( _mm_castps_si128(alp), 32 ) ), STBIR_onezeros )
1412-
1413- #define stbir__simdf_swiz( reg, one, two, three, four ) _mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( reg ), (one<<0) + (two<<2) + (three<<4) + (four<<6) ) )
1414-
1415- #define stbir__simdi_and( out, reg0, reg1 ) (out) = _mm_and_si128( reg0, reg1 )
1416- #define stbir__simdi_or( out, reg0, reg1 ) (out) = _mm_or_si128( reg0, reg1 )
1417- #define stbir__simdi_16madd( out, reg0, reg1 ) (out) = _mm_madd_epi16( reg0, reg1 )
1418-
1419- #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
1420- { \
1421- stbir__simdf af,bf; \
1422- stbir__simdi a,b; \
1423- af = _mm_min_ps( aa, STBIR_max_uint8_as_float ); \
1424- bf = _mm_min_ps( bb, STBIR_max_uint8_as_float ); \
1425- af = _mm_max_ps( af, _mm_setzero_ps() ); \
1426- bf = _mm_max_ps( bf, _mm_setzero_ps() ); \
1427- a = _mm_cvttps_epi32( af ); \
1428- b = _mm_cvttps_epi32( bf ); \
1429- a = _mm_packs_epi32( a, b ); \
1430- out = _mm_packus_epi16( a, a ); \
1431- }
1432-
1433- #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
1434- stbir__simdf_load( o0, (ptr) ); \
1435- stbir__simdf_load( o1, (ptr)+4 ); \
1436- stbir__simdf_load( o2, (ptr)+8 ); \
1437- stbir__simdf_load( o3, (ptr)+12 ); \
1438- { \
1439- __m128 tmp0, tmp1, tmp2, tmp3; \
1440- tmp0 = _mm_unpacklo_ps(o0, o1); \
1441- tmp2 = _mm_unpacklo_ps(o2, o3); \
1442- tmp1 = _mm_unpackhi_ps(o0, o1); \
1443- tmp3 = _mm_unpackhi_ps(o2, o3); \
1444- o0 = _mm_movelh_ps(tmp0, tmp2); \
1445- o1 = _mm_movehl_ps(tmp2, tmp0); \
1446- o2 = _mm_movelh_ps(tmp1, tmp3); \
1447- o3 = _mm_movehl_ps(tmp3, tmp1); \
1448- }
1449-
1450- #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
1451- r0 = _mm_packs_epi32( r0, r1 ); \
1452- r2 = _mm_packs_epi32( r2, r3 ); \
1453- r1 = _mm_unpacklo_epi16( r0, r2 ); \
1454- r3 = _mm_unpackhi_epi16( r0, r2 ); \
1455- r0 = _mm_unpacklo_epi16( r1, r3 ); \
1456- r2 = _mm_unpackhi_epi16( r1, r3 ); \
1457- r0 = _mm_packus_epi16( r0, r2 ); \
1458- stbir__simdi_store( ptr, r0 ); \
1459-
1460- #define stbir__simdi_32shr( out, reg, imm ) out = _mm_srli_epi32( reg, imm )
1461-
1462- #if defined(_MSC_VER) && !defined(__clang__)
1463- // msvc inits with 8 bytes
1464- #define STBIR__CONST_32_TO_8( v ) (char)(unsigned char)((v)&255),(char)(unsigned char)(((v)>>8)&255),(char)(unsigned char)(((v)>>16)&255),(char)(unsigned char)(((v)>>24)&255)
1465- #define STBIR__CONST_4_32i( v ) STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v )
1466- #define STBIR__CONST_4d_32i( v0, v1, v2, v3 ) STBIR__CONST_32_TO_8( v0 ), STBIR__CONST_32_TO_8( v1 ), STBIR__CONST_32_TO_8( v2 ), STBIR__CONST_32_TO_8( v3 )
1467- #else
1468- // everything else inits with long long's
1469- #define STBIR__CONST_4_32i( v ) (long long)((((stbir_uint64)(stbir_uint32)(v))<<32)|((stbir_uint64)(stbir_uint32)(v))),(long long)((((stbir_uint64)(stbir_uint32)(v))<<32)|((stbir_uint64)(stbir_uint32)(v)))
1470- #define STBIR__CONST_4d_32i( v0, v1, v2, v3 ) (long long)((((stbir_uint64)(stbir_uint32)(v1))<<32)|((stbir_uint64)(stbir_uint32)(v0))),(long long)((((stbir_uint64)(stbir_uint32)(v3))<<32)|((stbir_uint64)(stbir_uint32)(v2)))
1471- #endif
1472-
1473- #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = { x, x, x, x }
1474- #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { STBIR__CONST_4_32i(x) }
1475- #define STBIR__CONSTF(var) (var)
1476- #define STBIR__CONSTI(var) (var)
1477-
1478- #if defined(STBIR_AVX) || defined(__SSE4_1__)
1479- #include <smmintrin.h>
1480- #define stbir__simdf_pack_to_8words(out,reg0,reg1) out = _mm_packus_epi32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg0,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())), _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg1,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())))
1481- #else
1482- static STBIR__SIMDI_CONST(stbir__s32_32768, 32768);
1483- static STBIR__SIMDI_CONST(stbir__s16_32768, ((32768<<16)|32768));
1484-
1485- #define stbir__simdf_pack_to_8words(out,reg0,reg1) \
1486- { \
1487- stbir__simdi tmp0,tmp1; \
1488- tmp0 = _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg0,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())); \
1489- tmp1 = _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg1,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())); \
1490- tmp0 = _mm_sub_epi32( tmp0, stbir__s32_32768 ); \
1491- tmp1 = _mm_sub_epi32( tmp1, stbir__s32_32768 ); \
1492- out = _mm_packs_epi32( tmp0, tmp1 ); \
1493- out = _mm_sub_epi16( out, stbir__s16_32768 ); \
1494- }
1495-
1496- #endif
1497-
1498- #define STBIR_SIMD
1499-
1500- // if we detect AVX, set the simd8 defines
1501- #ifdef STBIR_AVX
1502- #include <immintrin.h>
1503- #define STBIR_SIMD8
1504- #define stbir__simdf8 __m256
1505- #define stbir__simdi8 __m256i
1506- #define stbir__simdf8_load( out, ptr ) (out) = _mm256_loadu_ps( (float const *)(ptr) )
1507- #define stbir__simdi8_load( out, ptr ) (out) = _mm256_loadu_si256( (__m256i const *)(ptr) )
1508- #define stbir__simdf8_mult( out, a, b ) (out) = _mm256_mul_ps( (a), (b) )
1509- #define stbir__simdf8_store( ptr, out ) _mm256_storeu_ps( (float*)(ptr), out )
1510- #define stbir__simdi8_store( ptr, reg ) _mm256_storeu_si256( (__m256i*)(ptr), reg )
1511- #define stbir__simdf8_frep8( fval ) _mm256_set1_ps( fval )
1512-
1513- #define stbir__simdf8_min( out, reg0, reg1 ) (out) = _mm256_min_ps( reg0, reg1 )
1514- #define stbir__simdf8_max( out, reg0, reg1 ) (out) = _mm256_max_ps( reg0, reg1 )
1515-
1516- #define stbir__simdf8_add4halves( out, bot4, top8 ) (out) = _mm_add_ps( bot4, _mm256_extractf128_ps( top8, 1 ) )
1517- #define stbir__simdf8_mult_mem( out, reg, ptr ) (out) = _mm256_mul_ps( reg, _mm256_loadu_ps( (float const*)(ptr) ) )
1518- #define stbir__simdf8_add_mem( out, reg, ptr ) (out) = _mm256_add_ps( reg, _mm256_loadu_ps( (float const*)(ptr) ) )
1519- #define stbir__simdf8_add( out, a, b ) (out) = _mm256_add_ps( a, b )
1520- #define stbir__simdf8_load1b( out, ptr ) (out) = _mm256_broadcast_ss( ptr )
1521- #define stbir__simdf_load1rep4( out, ptr ) (out) = _mm_broadcast_ss( ptr ) // avx load instruction
1522-
1523- #define stbir__simdi8_convert_i32_to_float(out, ireg) (out) = _mm256_cvtepi32_ps( ireg )
1524- #define stbir__simdf8_convert_float_to_i32( i, f ) (i) = _mm256_cvttps_epi32(f)
1525-
1526- #define stbir__simdf8_bot4s( out, a, b ) (out) = _mm256_permute2f128_ps(a,b, (0<<0)+(2<<4) )
1527- #define stbir__simdf8_top4s( out, a, b ) (out) = _mm256_permute2f128_ps(a,b, (1<<0)+(3<<4) )
1528-
1529- #define stbir__simdf8_gettop4( reg ) _mm256_extractf128_ps(reg,1)
1530-
1531- #ifdef STBIR_AVX2
1532-
1533- #define stbir__simdi8_expand_u8_to_u32(out0,out1,ireg) \
1534- { \
1535- stbir__simdi8 a, zero =_mm256_setzero_si256();\
1536- a = _mm256_permute4x64_epi64( _mm256_unpacklo_epi8( _mm256_permute4x64_epi64(_mm256_castsi128_si256(ireg),(0<<0)+(2<<2)+(1<<4)+(3<<6)), zero ),(0<<0)+(2<<2)+(1<<4)+(3<<6)); \
1537- out0 = _mm256_unpacklo_epi16( a, zero ); \
1538- out1 = _mm256_unpackhi_epi16( a, zero ); \
1539- }
1540-
1541- #define stbir__simdf8_pack_to_16bytes(out,aa,bb) \
1542- { \
1543- stbir__simdi8 t; \
1544- stbir__simdf8 af,bf; \
1545- stbir__simdi8 a,b; \
1546- af = _mm256_min_ps( aa, STBIR_max_uint8_as_floatX ); \
1547- bf = _mm256_min_ps( bb, STBIR_max_uint8_as_floatX ); \
1548- af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1549- bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1550- a = _mm256_cvttps_epi32( af ); \
1551- b = _mm256_cvttps_epi32( bf ); \
1552- t = _mm256_permute4x64_epi64( _mm256_packs_epi32( a, b ), (0<<0)+(2<<2)+(1<<4)+(3<<6) ); \
1553- out = _mm256_castsi256_si128( _mm256_permute4x64_epi64( _mm256_packus_epi16( t, t ), (0<<0)+(2<<2)+(1<<4)+(3<<6) ) ); \
1554- }
1555-
1556- #define stbir__simdi8_expand_u16_to_u32(out,ireg) out = _mm256_unpacklo_epi16( _mm256_permute4x64_epi64(_mm256_castsi128_si256(ireg),(0<<0)+(2<<2)+(1<<4)+(3<<6)), _mm256_setzero_si256() );
1557-
1558- #define stbir__simdf8_pack_to_16words(out,aa,bb) \
1559- { \
1560- stbir__simdf8 af,bf; \
1561- stbir__simdi8 a,b; \
1562- af = _mm256_min_ps( aa, STBIR_max_uint16_as_floatX ); \
1563- bf = _mm256_min_ps( bb, STBIR_max_uint16_as_floatX ); \
1564- af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1565- bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1566- a = _mm256_cvttps_epi32( af ); \
1567- b = _mm256_cvttps_epi32( bf ); \
1568- (out) = _mm256_permute4x64_epi64( _mm256_packus_epi32(a, b), (0<<0)+(2<<2)+(1<<4)+(3<<6) ); \
1569- }
1570-
1571- #else
1572-
1573- #define stbir__simdi8_expand_u8_to_u32(out0,out1,ireg) \
1574- { \
1575- stbir__simdi a,zero = _mm_setzero_si128(); \
1576- a = _mm_unpacklo_epi8( ireg, zero ); \
1577- out0 = _mm256_setr_m128i( _mm_unpacklo_epi16( a, zero ), _mm_unpackhi_epi16( a, zero ) ); \
1578- a = _mm_unpackhi_epi8( ireg, zero ); \
1579- out1 = _mm256_setr_m128i( _mm_unpacklo_epi16( a, zero ), _mm_unpackhi_epi16( a, zero ) ); \
1580- }
1581-
1582- #define stbir__simdf8_pack_to_16bytes(out,aa,bb) \
1583- { \
1584- stbir__simdi t; \
1585- stbir__simdf8 af,bf; \
1586- stbir__simdi8 a,b; \
1587- af = _mm256_min_ps( aa, STBIR_max_uint8_as_floatX ); \
1588- bf = _mm256_min_ps( bb, STBIR_max_uint8_as_floatX ); \
1589- af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1590- bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1591- a = _mm256_cvttps_epi32( af ); \
1592- b = _mm256_cvttps_epi32( bf ); \
1593- out = _mm_packs_epi32( _mm256_castsi256_si128(a), _mm256_extractf128_si256( a, 1 ) ); \
1594- out = _mm_packus_epi16( out, out ); \
1595- t = _mm_packs_epi32( _mm256_castsi256_si128(b), _mm256_extractf128_si256( b, 1 ) ); \
1596- t = _mm_packus_epi16( t, t ); \
1597- out = _mm_castps_si128( _mm_shuffle_ps( _mm_castsi128_ps(out), _mm_castsi128_ps(t), (0<<0)+(1<<2)+(0<<4)+(1<<6) ) ); \
1598- }
1599-
1600- #define stbir__simdi8_expand_u16_to_u32(out,ireg) \
1601- { \
1602- stbir__simdi a,b,zero = _mm_setzero_si128(); \
1603- a = _mm_unpacklo_epi16( ireg, zero ); \
1604- b = _mm_unpackhi_epi16( ireg, zero ); \
1605- out = _mm256_insertf128_si256( _mm256_castsi128_si256( a ), b, 1 ); \
1606- }
1607-
1608- #define stbir__simdf8_pack_to_16words(out,aa,bb) \
1609- { \
1610- stbir__simdi t0,t1; \
1611- stbir__simdf8 af,bf; \
1612- stbir__simdi8 a,b; \
1613- af = _mm256_min_ps( aa, STBIR_max_uint16_as_floatX ); \
1614- bf = _mm256_min_ps( bb, STBIR_max_uint16_as_floatX ); \
1615- af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1616- bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1617- a = _mm256_cvttps_epi32( af ); \
1618- b = _mm256_cvttps_epi32( bf ); \
1619- t0 = _mm_packus_epi32( _mm256_castsi256_si128(a), _mm256_extractf128_si256( a, 1 ) ); \
1620- t1 = _mm_packus_epi32( _mm256_castsi256_si128(b), _mm256_extractf128_si256( b, 1 ) ); \
1621- out = _mm256_setr_m128i( t0, t1 ); \
1622- }
1623-
1624- #endif
1625-
1626- static __m256i stbir_00001111 = { STBIR__CONST_4d_32i( 0, 0, 0, 0 ), STBIR__CONST_4d_32i( 1, 1, 1, 1 ) };
1627- #define stbir__simdf8_0123to00001111( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_00001111 )
1628-
1629- static __m256i stbir_22223333 = { STBIR__CONST_4d_32i( 2, 2, 2, 2 ), STBIR__CONST_4d_32i( 3, 3, 3, 3 ) };
1630- #define stbir__simdf8_0123to22223333( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_22223333 )
1631-
1632- #define stbir__simdf8_0123to2222( out, in ) (out) = stbir__simdf_swiz(_mm256_castps256_ps128(in), 2,2,2,2 )
1633-
1634- #define stbir__simdf8_load4b( out, ptr ) (out) = _mm256_broadcast_ps( (__m128 const *)(ptr) )
1635-
1636- static __m256i stbir_00112233 = { STBIR__CONST_4d_32i( 0, 0, 1, 1 ), STBIR__CONST_4d_32i( 2, 2, 3, 3 ) };
1637- #define stbir__simdf8_0123to00112233( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_00112233 )
1638- #define stbir__simdf8_add4( out, a8, b ) (out) = _mm256_add_ps( a8, _mm256_castps128_ps256( b ) )
1639-
1640- static __m256i stbir_load6 = { STBIR__CONST_4_32i( 0x80000000 ), STBIR__CONST_4d_32i( 0x80000000, 0x80000000, 0, 0 ) };
1641- #define stbir__simdf8_load6z( out, ptr ) (out) = _mm256_maskload_ps( ptr, stbir_load6 )
1642-
1643- #define stbir__simdf8_0123to00000000( out, in ) (out) = _mm256_shuffle_ps ( in, in, (0<<0)+(0<<2)+(0<<4)+(0<<6) )
1644- #define stbir__simdf8_0123to11111111( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(1<<2)+(1<<4)+(1<<6) )
1645- #define stbir__simdf8_0123to22222222( out, in ) (out) = _mm256_shuffle_ps ( in, in, (2<<0)+(2<<2)+(2<<4)+(2<<6) )
1646- #define stbir__simdf8_0123to33333333( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(3<<2)+(3<<4)+(3<<6) )
1647- #define stbir__simdf8_0123to21032103( out, in ) (out) = _mm256_shuffle_ps ( in, in, (2<<0)+(1<<2)+(0<<4)+(3<<6) )
1648- #define stbir__simdf8_0123to32103210( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(2<<2)+(1<<4)+(0<<6) )
1649- #define stbir__simdf8_0123to12301230( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(2<<2)+(3<<4)+(0<<6) )
1650- #define stbir__simdf8_0123to10321032( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(0<<2)+(3<<4)+(2<<6) )
1651- #define stbir__simdf8_0123to30123012( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(0<<2)+(1<<4)+(2<<6) )
1652-
1653- #define stbir__simdf8_0123to11331133( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(1<<2)+(3<<4)+(3<<6) )
1654- #define stbir__simdf8_0123to00220022( out, in ) (out) = _mm256_shuffle_ps ( in, in, (0<<0)+(0<<2)+(2<<4)+(2<<6) )
1655-
1656- #define stbir__simdf8_aaa1( out, alp, ones ) (out) = _mm256_blend_ps( alp, ones, (1<<0)+(1<<1)+(1<<2)+(0<<3)+(1<<4)+(1<<5)+(1<<6)+(0<<7)); (out)=_mm256_shuffle_ps( out,out, (3<<0) + (3<<2) + (3<<4) + (0<<6) )
1657- #define stbir__simdf8_1aaa( out, alp, ones ) (out) = _mm256_blend_ps( alp, ones, (0<<0)+(1<<1)+(1<<2)+(1<<3)+(0<<4)+(1<<5)+(1<<6)+(1<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (0<<4) + (0<<6) )
1658- #define stbir__simdf8_a1a1( out, alp, ones) (out) = _mm256_blend_ps( alp, ones, (1<<0)+(0<<1)+(1<<2)+(0<<3)+(1<<4)+(0<<5)+(1<<6)+(0<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (3<<4) + (2<<6) )
1659- #define stbir__simdf8_1a1a( out, alp, ones) (out) = _mm256_blend_ps( alp, ones, (0<<0)+(1<<1)+(0<<2)+(1<<3)+(0<<4)+(1<<5)+(0<<6)+(1<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (3<<4) + (2<<6) )
1660-
1661- #define stbir__simdf8_zero( reg ) (reg) = _mm256_setzero_ps()
1662-
1663- #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd
1664- #define stbir__simdf8_madd( out, add, mul1, mul2 ) (out) = _mm256_fmadd_ps( mul1, mul2, add )
1665- #define stbir__simdf8_madd_mem( out, add, mul, ptr ) (out) = _mm256_fmadd_ps( mul, _mm256_loadu_ps( (float const*)(ptr) ), add )
1666- #define stbir__simdf8_madd_mem4( out, add, mul, ptr )(out) = _mm256_fmadd_ps( _mm256_setr_m128( mul, _mm_setzero_ps() ), _mm256_setr_m128( _mm_loadu_ps( (float const*)(ptr) ), _mm_setzero_ps() ), add )
1667- #else
1668- #define stbir__simdf8_madd( out, add, mul1, mul2 ) (out) = _mm256_add_ps( add, _mm256_mul_ps( mul1, mul2 ) )
1669- #define stbir__simdf8_madd_mem( out, add, mul, ptr ) (out) = _mm256_add_ps( add, _mm256_mul_ps( mul, _mm256_loadu_ps( (float const*)(ptr) ) ) )
1670- #define stbir__simdf8_madd_mem4( out, add, mul, ptr ) (out) = _mm256_add_ps( add, _mm256_setr_m128( _mm_mul_ps( mul, _mm_loadu_ps( (float const*)(ptr) ) ), _mm_setzero_ps() ) )
1671- #endif
1672- #define stbir__if_simdf8_cast_to_simdf4( val ) _mm256_castps256_ps128( val )
1673-
1674- #endif
1675-
1676- #ifdef STBIR_FLOORF
1677- #undef STBIR_FLOORF
1678- #endif
1679- #define STBIR_FLOORF stbir_simd_floorf
1680- static stbir__inline float stbir_simd_floorf(float x) // martins floorf
1681- {
1682- #if defined(STBIR_AVX) || defined(__SSE4_1__) || defined(STBIR_SSE41)
1683- __m128 t = _mm_set_ss(x);
1684- return _mm_cvtss_f32( _mm_floor_ss(t, t) );
1685- #else
1686- __m128 f = _mm_set_ss(x);
1687- __m128 t = _mm_cvtepi32_ps(_mm_cvttps_epi32(f));
1688- __m128 r = _mm_add_ss(t, _mm_and_ps(_mm_cmplt_ss(f, t), _mm_set_ss(-1.0f)));
1689- return _mm_cvtss_f32(r);
1690- #endif
1691- }
1692-
1693- #ifdef STBIR_CEILF
1694- #undef STBIR_CEILF
1695- #endif
1696- #define STBIR_CEILF stbir_simd_ceilf
1697- static stbir__inline float stbir_simd_ceilf(float x) // martins ceilf
1698- {
1699- #if defined(STBIR_AVX) || defined(__SSE4_1__) || defined(STBIR_SSE41)
1700- __m128 t = _mm_set_ss(x);
1701- return _mm_cvtss_f32( _mm_ceil_ss(t, t) );
1702- #else
1703- __m128 f = _mm_set_ss(x);
1704- __m128 t = _mm_cvtepi32_ps(_mm_cvttps_epi32(f));
1705- __m128 r = _mm_add_ss(t, _mm_and_ps(_mm_cmplt_ss(t, f), _mm_set_ss(1.0f)));
1706- return _mm_cvtss_f32(r);
1707- #endif
1708- }
1709-
1710-#elif defined(STBIR_NEON)
1711-
1712- #include <arm_neon.h>
1713-
1714- #define stbir__simdf float32x4_t
1715- #define stbir__simdi uint32x4_t
1716-
1717- #define stbir_simdi_castf( reg ) vreinterpretq_u32_f32(reg)
1718- #define stbir_simdf_casti( reg ) vreinterpretq_f32_u32(reg)
1719-
1720- #define stbir__simdf_load( reg, ptr ) (reg) = vld1q_f32( (float const*)(ptr) )
1721- #define stbir__simdi_load( reg, ptr ) (reg) = vld1q_u32( (uint32_t const*)(ptr) )
1722- #define stbir__simdf_load1( out, ptr ) (out) = vld1q_dup_f32( (float const*)(ptr) ) // top values can be random (not denormal or nan for perf)
1723- #define stbir__simdi_load1( out, ptr ) (out) = vld1q_dup_u32( (uint32_t const*)(ptr) )
1724- #define stbir__simdf_load1z( out, ptr ) (out) = vld1q_lane_f32( (float const*)(ptr), vdupq_n_f32(0), 0 ) // top values must be zero
1725- #define stbir__simdf_frep4( fvar ) vdupq_n_f32( fvar )
1726- #define stbir__simdf_load1frep4( out, fvar ) (out) = vdupq_n_f32( fvar )
1727- #define stbir__simdf_load2( out, ptr ) (out) = vcombine_f32( vld1_f32( (float const*)(ptr) ), vcreate_f32(0) ) // top values can be random (not denormal or nan for perf)
1728- #define stbir__simdf_load2z( out, ptr ) (out) = vcombine_f32( vld1_f32( (float const*)(ptr) ), vcreate_f32(0) ) // top values must be zero
1729- #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = vcombine_f32( vget_low_f32(reg), vld1_f32( (float const*)(ptr) ) )
1730-
1731- #define stbir__simdf_zeroP() vdupq_n_f32(0)
1732- #define stbir__simdf_zero( reg ) (reg) = vdupq_n_f32(0)
1733-
1734- #define stbir__simdf_store( ptr, reg ) vst1q_f32( (float*)(ptr), reg )
1735- #define stbir__simdf_store1( ptr, reg ) vst1q_lane_f32( (float*)(ptr), reg, 0)
1736- #define stbir__simdf_store2( ptr, reg ) vst1_f32( (float*)(ptr), vget_low_f32(reg) )
1737- #define stbir__simdf_store2h( ptr, reg ) vst1_f32( (float*)(ptr), vget_high_f32(reg) )
1738-
1739- #define stbir__simdi_store( ptr, reg ) vst1q_u32( (uint32_t*)(ptr), reg )
1740- #define stbir__simdi_store1( ptr, reg ) vst1q_lane_u32( (uint32_t*)(ptr), reg, 0 )
1741- #define stbir__simdi_store2( ptr, reg ) vst1_u32( (uint32_t*)(ptr), vget_low_u32(reg) )
1742-
1743- #define stbir__prefetch( ptr )
1744-
1745- #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
1746- { \
1747- uint16x8_t l = vmovl_u8( vget_low_u8 ( vreinterpretq_u8_u32(ireg) ) ); \
1748- uint16x8_t h = vmovl_u8( vget_high_u8( vreinterpretq_u8_u32(ireg) ) ); \
1749- out0 = vmovl_u16( vget_low_u16 ( l ) ); \
1750- out1 = vmovl_u16( vget_high_u16( l ) ); \
1751- out2 = vmovl_u16( vget_low_u16 ( h ) ); \
1752- out3 = vmovl_u16( vget_high_u16( h ) ); \
1753- }
1754-
1755- #define stbir__simdi_expand_u8_to_1u32(out,ireg) \
1756- { \
1757- uint16x8_t tmp = vmovl_u8( vget_low_u8( vreinterpretq_u8_u32(ireg) ) ); \
1758- out = vmovl_u16( vget_low_u16( tmp ) ); \
1759- }
1760-
1761- #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
1762- { \
1763- uint16x8_t tmp = vreinterpretq_u16_u32(ireg); \
1764- out0 = vmovl_u16( vget_low_u16 ( tmp ) ); \
1765- out1 = vmovl_u16( vget_high_u16( tmp ) ); \
1766- }
1767-
1768- #define stbir__simdf_convert_float_to_i32( i, f ) (i) = vreinterpretq_u32_s32( vcvtq_s32_f32(f) )
1769- #define stbir__simdf_convert_float_to_int( f ) vgetq_lane_s32(vcvtq_s32_f32(f), 0)
1770- #define stbir__simdi_to_int( i ) (int)vgetq_lane_u32(i, 0)
1771- #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)vgetq_lane_s32(vcvtq_s32_f32(vmaxq_f32(vminq_f32(f,STBIR__CONSTF(STBIR_max_uint8_as_float)),vdupq_n_f32(0))), 0))
1772- #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)vgetq_lane_s32(vcvtq_s32_f32(vmaxq_f32(vminq_f32(f,STBIR__CONSTF(STBIR_max_uint16_as_float)),vdupq_n_f32(0))), 0))
1773- #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = vcvtq_f32_s32( vreinterpretq_s32_u32(ireg) )
1774- #define stbir__simdf_add( out, reg0, reg1 ) (out) = vaddq_f32( reg0, reg1 )
1775- #define stbir__simdf_mult( out, reg0, reg1 ) (out) = vmulq_f32( reg0, reg1 )
1776- #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = vmulq_f32( reg, vld1q_f32( (float const*)(ptr) ) )
1777- #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = vmulq_f32( reg, vld1q_dup_f32( (float const*)(ptr) ) )
1778- #define stbir__simdf_add_mem( out, reg, ptr ) (out) = vaddq_f32( reg, vld1q_f32( (float const*)(ptr) ) )
1779- #define stbir__simdf_add1_mem( out, reg, ptr ) (out) = vaddq_f32( reg, vld1q_dup_f32( (float const*)(ptr) ) )
1780-
1781- #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd (and also x64 no madd to arm madd)
1782- #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = vfmaq_f32( add, mul1, mul2 )
1783- #define stbir__simdf_madd1( out, add, mul1, mul2 ) (out) = vfmaq_f32( add, mul1, mul2 )
1784- #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = vfmaq_f32( add, mul, vld1q_f32( (float const*)(ptr) ) )
1785- #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = vfmaq_f32( add, mul, vld1q_dup_f32( (float const*)(ptr) ) )
1786- #else
1787- #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = vaddq_f32( add, vmulq_f32( mul1, mul2 ) )
1788- #define stbir__simdf_madd1( out, add, mul1, mul2 ) (out) = vaddq_f32( add, vmulq_f32( mul1, mul2 ) )
1789- #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = vaddq_f32( add, vmulq_f32( mul, vld1q_f32( (float const*)(ptr) ) ) )
1790- #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = vaddq_f32( add, vmulq_f32( mul, vld1q_dup_f32( (float const*)(ptr) ) ) )
1791- #endif
1792-
1793- #define stbir__simdf_add1( out, reg0, reg1 ) (out) = vaddq_f32( reg0, reg1 )
1794- #define stbir__simdf_mult1( out, reg0, reg1 ) (out) = vmulq_f32( reg0, reg1 )
1795-
1796- #define stbir__simdf_and( out, reg0, reg1 ) (out) = vreinterpretq_f32_u32( vandq_u32( vreinterpretq_u32_f32(reg0), vreinterpretq_u32_f32(reg1) ) )
1797- #define stbir__simdf_or( out, reg0, reg1 ) (out) = vreinterpretq_f32_u32( vorrq_u32( vreinterpretq_u32_f32(reg0), vreinterpretq_u32_f32(reg1) ) )
1798-
1799- #define stbir__simdf_min( out, reg0, reg1 ) (out) = vminq_f32( reg0, reg1 )
1800- #define stbir__simdf_max( out, reg0, reg1 ) (out) = vmaxq_f32( reg0, reg1 )
1801- #define stbir__simdf_min1( out, reg0, reg1 ) (out) = vminq_f32( reg0, reg1 )
1802- #define stbir__simdf_max1( out, reg0, reg1 ) (out) = vmaxq_f32( reg0, reg1 )
1803-
1804- #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out) = vextq_f32( reg0, reg1, 3 )
1805- #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out) = vextq_f32( reg0, reg1, 2 )
1806-
1807- #define stbir__simdf_a1a1( out, alp, ones ) (out) = vzipq_f32(vuzpq_f32(alp, alp).val[1], ones).val[0]
1808- #define stbir__simdf_1a1a( out, alp, ones ) (out) = vzipq_f32(ones, vuzpq_f32(alp, alp).val[0]).val[0]
1809-
1810- #if defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ )
1811-
1812- #define stbir__simdf_aaa1( out, alp, ones ) (out) = vcopyq_laneq_f32(vdupq_n_f32(vgetq_lane_f32(alp, 3)), 3, ones, 3)
1813- #define stbir__simdf_1aaa( out, alp, ones ) (out) = vcopyq_laneq_f32(vdupq_n_f32(vgetq_lane_f32(alp, 0)), 0, ones, 0)
1814-
1815- #if defined( _MSC_VER ) && !defined(__clang__)
1816- #define stbir_make16(a,b,c,d) vcombine_u8( \
1817- vcreate_u8( (4*a+0) | ((4*a+1)<<8) | ((4*a+2)<<16) | ((4*a+3)<<24) | \
1818- ((stbir_uint64)(4*b+0)<<32) | ((stbir_uint64)(4*b+1)<<40) | ((stbir_uint64)(4*b+2)<<48) | ((stbir_uint64)(4*b+3)<<56)), \
1819- vcreate_u8( (4*c+0) | ((4*c+1)<<8) | ((4*c+2)<<16) | ((4*c+3)<<24) | \
1820- ((stbir_uint64)(4*d+0)<<32) | ((stbir_uint64)(4*d+1)<<40) | ((stbir_uint64)(4*d+2)<<48) | ((stbir_uint64)(4*d+3)<<56) ) )
1821-
1822- static stbir__inline uint8x16x2_t stbir_make16x2(float32x4_t rega,float32x4_t regb)
1823- {
1824- uint8x16x2_t r = { vreinterpretq_u8_f32(rega), vreinterpretq_u8_f32(regb) };
1825- return r;
1826- }
1827- #else
1828- #define stbir_make16(a,b,c,d) (uint8x16_t){4*a+0,4*a+1,4*a+2,4*a+3,4*b+0,4*b+1,4*b+2,4*b+3,4*c+0,4*c+1,4*c+2,4*c+3,4*d+0,4*d+1,4*d+2,4*d+3}
1829- #define stbir_make16x2(a,b) (uint8x16x2_t){{vreinterpretq_u8_f32(a),vreinterpretq_u8_f32(b)}}
1830- #endif
1831-
1832- #define stbir__simdf_swiz( reg, one, two, three, four ) vreinterpretq_f32_u8( vqtbl1q_u8( vreinterpretq_u8_f32(reg), stbir_make16(one, two, three, four) ) )
1833- #define stbir__simdf_swiz2( rega, regb, one, two, three, four ) vreinterpretq_f32_u8( vqtbl2q_u8( stbir_make16x2(rega,regb), stbir_make16(one, two, three, four) ) )
1834-
1835- #define stbir__simdi_16madd( out, reg0, reg1 ) \
1836- { \
1837- int16x8_t r0 = vreinterpretq_s16_u32(reg0); \
1838- int16x8_t r1 = vreinterpretq_s16_u32(reg1); \
1839- int32x4_t tmp0 = vmull_s16( vget_low_s16(r0), vget_low_s16(r1) ); \
1840- int32x4_t tmp1 = vmull_s16( vget_high_s16(r0), vget_high_s16(r1) ); \
1841- (out) = vreinterpretq_u32_s32( vpaddq_s32(tmp0, tmp1) ); \
1842- }
1843-
1844- #else
1845-
1846- #define stbir__simdf_aaa1( out, alp, ones ) (out) = vsetq_lane_f32(1.0f, vdupq_n_f32(vgetq_lane_f32(alp, 3)), 3)
1847- #define stbir__simdf_1aaa( out, alp, ones ) (out) = vsetq_lane_f32(1.0f, vdupq_n_f32(vgetq_lane_f32(alp, 0)), 0)
1848-
1849- #if defined( _MSC_VER ) && !defined(__clang__)
1850- static stbir__inline uint8x8x2_t stbir_make8x2(float32x4_t reg)
1851- {
1852- uint8x8x2_t r = { { vget_low_u8(vreinterpretq_u8_f32(reg)), vget_high_u8(vreinterpretq_u8_f32(reg)) } };
1853- return r;
1854- }
1855- #define stbir_make8(a,b) vcreate_u8( \
1856- (4*a+0) | ((4*a+1)<<8) | ((4*a+2)<<16) | ((4*a+3)<<24) | \
1857- ((stbir_uint64)(4*b+0)<<32) | ((stbir_uint64)(4*b+1)<<40) | ((stbir_uint64)(4*b+2)<<48) | ((stbir_uint64)(4*b+3)<<56) )
1858- #else
1859- #define stbir_make8x2(reg) (uint8x8x2_t){ { vget_low_u8(vreinterpretq_u8_f32(reg)), vget_high_u8(vreinterpretq_u8_f32(reg)) } }
1860- #define stbir_make8(a,b) (uint8x8_t){4*a+0,4*a+1,4*a+2,4*a+3,4*b+0,4*b+1,4*b+2,4*b+3}
1861- #endif
1862-
1863- #define stbir__simdf_swiz( reg, one, two, three, four ) vreinterpretq_f32_u8( vcombine_u8( \
1864- vtbl2_u8( stbir_make8x2( reg ), stbir_make8( one, two ) ), \
1865- vtbl2_u8( stbir_make8x2( reg ), stbir_make8( three, four ) ) ) )
1866-
1867- #define stbir__simdi_16madd( out, reg0, reg1 ) \
1868- { \
1869- int16x8_t r0 = vreinterpretq_s16_u32(reg0); \
1870- int16x8_t r1 = vreinterpretq_s16_u32(reg1); \
1871- int32x4_t tmp0 = vmull_s16( vget_low_s16(r0), vget_low_s16(r1) ); \
1872- int32x4_t tmp1 = vmull_s16( vget_high_s16(r0), vget_high_s16(r1) ); \
1873- int32x2_t out0 = vpadd_s32( vget_low_s32(tmp0), vget_high_s32(tmp0) ); \
1874- int32x2_t out1 = vpadd_s32( vget_low_s32(tmp1), vget_high_s32(tmp1) ); \
1875- (out) = vreinterpretq_u32_s32( vcombine_s32(out0, out1) ); \
1876- }
1877-
1878- #endif
1879-
1880- #define stbir__simdi_and( out, reg0, reg1 ) (out) = vandq_u32( reg0, reg1 )
1881- #define stbir__simdi_or( out, reg0, reg1 ) (out) = vorrq_u32( reg0, reg1 )
1882-
1883- #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
1884- { \
1885- float32x4_t af = vmaxq_f32( vminq_f32(aa,STBIR__CONSTF(STBIR_max_uint8_as_float) ), vdupq_n_f32(0) ); \
1886- float32x4_t bf = vmaxq_f32( vminq_f32(bb,STBIR__CONSTF(STBIR_max_uint8_as_float) ), vdupq_n_f32(0) ); \
1887- int16x4_t ai = vqmovn_s32( vcvtq_s32_f32( af ) ); \
1888- int16x4_t bi = vqmovn_s32( vcvtq_s32_f32( bf ) ); \
1889- uint8x8_t out8 = vqmovun_s16( vcombine_s16(ai, bi) ); \
1890- out = vreinterpretq_u32_u8( vcombine_u8(out8, out8) ); \
1891- }
1892-
1893- #define stbir__simdf_pack_to_8words(out,aa,bb) \
1894- { \
1895- float32x4_t af = vmaxq_f32( vminq_f32(aa,STBIR__CONSTF(STBIR_max_uint16_as_float) ), vdupq_n_f32(0) ); \
1896- float32x4_t bf = vmaxq_f32( vminq_f32(bb,STBIR__CONSTF(STBIR_max_uint16_as_float) ), vdupq_n_f32(0) ); \
1897- int32x4_t ai = vcvtq_s32_f32( af ); \
1898- int32x4_t bi = vcvtq_s32_f32( bf ); \
1899- out = vreinterpretq_u32_u16( vcombine_u16(vqmovun_s32(ai), vqmovun_s32(bi)) ); \
1900- }
1901-
1902- #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
1903- { \
1904- int16x4x2_t tmp0 = vzip_s16( vqmovn_s32(vreinterpretq_s32_u32(r0)), vqmovn_s32(vreinterpretq_s32_u32(r2)) ); \
1905- int16x4x2_t tmp1 = vzip_s16( vqmovn_s32(vreinterpretq_s32_u32(r1)), vqmovn_s32(vreinterpretq_s32_u32(r3)) ); \
1906- uint8x8x2_t out = \
1907- { { \
1908- vqmovun_s16( vcombine_s16(tmp0.val[0], tmp0.val[1]) ), \
1909- vqmovun_s16( vcombine_s16(tmp1.val[0], tmp1.val[1]) ), \
1910- } }; \
1911- vst2_u8(ptr, out); \
1912- }
1913-
1914- #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
1915- { \
1916- float32x4x4_t tmp = vld4q_f32(ptr); \
1917- o0 = tmp.val[0]; \
1918- o1 = tmp.val[1]; \
1919- o2 = tmp.val[2]; \
1920- o3 = tmp.val[3]; \
1921- }
1922-
1923- #define stbir__simdi_32shr( out, reg, imm ) out = vshrq_n_u32( reg, imm )
1924-
1925- #if defined( _MSC_VER ) && !defined(__clang__)
1926- #define STBIR__SIMDF_CONST(var, x) __declspec(align(8)) float var[] = { x, x, x, x }
1927- #define STBIR__SIMDI_CONST(var, x) __declspec(align(8)) uint32_t var[] = { x, x, x, x }
1928- #define STBIR__CONSTF(var) (*(const float32x4_t*)var)
1929- #define STBIR__CONSTI(var) (*(const uint32x4_t*)var)
1930- #else
1931- #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = { x, x, x, x }
1932- #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { x, x, x, x }
1933- #define STBIR__CONSTF(var) (var)
1934- #define STBIR__CONSTI(var) (var)
1935- #endif
1936-
1937- #ifdef STBIR_FLOORF
1938- #undef STBIR_FLOORF
1939- #endif
1940- #define STBIR_FLOORF stbir_simd_floorf
1941- static stbir__inline float stbir_simd_floorf(float x)
1942- {
1943- #if defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ )
1944- return vget_lane_f32( vrndm_f32( vdup_n_f32(x) ), 0);
1945- #else
1946- float32x2_t f = vdup_n_f32(x);
1947- float32x2_t t = vcvt_f32_s32(vcvt_s32_f32(f));
1948- uint32x2_t a = vclt_f32(f, t);
1949- uint32x2_t b = vreinterpret_u32_f32(vdup_n_f32(-1.0f));
1950- float32x2_t r = vadd_f32(t, vreinterpret_f32_u32(vand_u32(a, b)));
1951- return vget_lane_f32(r, 0);
1952- #endif
1953- }
1954-
1955- #ifdef STBIR_CEILF
1956- #undef STBIR_CEILF
1957- #endif
1958- #define STBIR_CEILF stbir_simd_ceilf
1959- static stbir__inline float stbir_simd_ceilf(float x)
1960- {
1961- #if defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ )
1962- return vget_lane_f32( vrndp_f32( vdup_n_f32(x) ), 0);
1963- #else
1964- float32x2_t f = vdup_n_f32(x);
1965- float32x2_t t = vcvt_f32_s32(vcvt_s32_f32(f));
1966- uint32x2_t a = vclt_f32(t, f);
1967- uint32x2_t b = vreinterpret_u32_f32(vdup_n_f32(1.0f));
1968- float32x2_t r = vadd_f32(t, vreinterpret_f32_u32(vand_u32(a, b)));
1969- return vget_lane_f32(r, 0);
1970- #endif
1971- }
1972-
1973- #define STBIR_SIMD
1974-
1975-#elif defined(STBIR_WASM)
1976-
1977- #include <wasm_simd128.h>
1978-
1979- #define stbir__simdf v128_t
1980- #define stbir__simdi v128_t
1981-
1982- #define stbir_simdi_castf( reg ) (reg)
1983- #define stbir_simdf_casti( reg ) (reg)
1984-
1985- #define stbir__simdf_load( reg, ptr ) (reg) = wasm_v128_load( (void const*)(ptr) )
1986- #define stbir__simdi_load( reg, ptr ) (reg) = wasm_v128_load( (void const*)(ptr) )
1987- #define stbir__simdf_load1( out, ptr ) (out) = wasm_v128_load32_splat( (void const*)(ptr) ) // top values can be random (not denormal or nan for perf)
1988- #define stbir__simdi_load1( out, ptr ) (out) = wasm_v128_load32_splat( (void const*)(ptr) )
1989- #define stbir__simdf_load1z( out, ptr ) (out) = wasm_v128_load32_zero( (void const*)(ptr) ) // top values must be zero
1990- #define stbir__simdf_frep4( fvar ) wasm_f32x4_splat( fvar )
1991- #define stbir__simdf_load1frep4( out, fvar ) (out) = wasm_f32x4_splat( fvar )
1992- #define stbir__simdf_load2( out, ptr ) (out) = wasm_v128_load64_splat( (void const*)(ptr) ) // top values can be random (not denormal or nan for perf)
1993- #define stbir__simdf_load2z( out, ptr ) (out) = wasm_v128_load64_zero( (void const*)(ptr) ) // top values must be zero
1994- #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = wasm_v128_load64_lane( (void const*)(ptr), reg, 1 )
1995-
1996- #define stbir__simdf_zeroP() wasm_f32x4_const_splat(0)
1997- #define stbir__simdf_zero( reg ) (reg) = wasm_f32x4_const_splat(0)
1998-
1999- #define stbir__simdf_store( ptr, reg ) wasm_v128_store( (void*)(ptr), reg )
2000- #define stbir__simdf_store1( ptr, reg ) wasm_v128_store32_lane( (void*)(ptr), reg, 0 )
2001- #define stbir__simdf_store2( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 0 )
2002- #define stbir__simdf_store2h( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 1 )
2003-
2004- #define stbir__simdi_store( ptr, reg ) wasm_v128_store( (void*)(ptr), reg )
2005- #define stbir__simdi_store1( ptr, reg ) wasm_v128_store32_lane( (void*)(ptr), reg, 0 )
2006- #define stbir__simdi_store2( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 0 )
2007-
2008- #define stbir__prefetch( ptr )
2009-
2010- #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
2011- { \
2012- v128_t l = wasm_u16x8_extend_low_u8x16 ( ireg ); \
2013- v128_t h = wasm_u16x8_extend_high_u8x16( ireg ); \
2014- out0 = wasm_u32x4_extend_low_u16x8 ( l ); \
2015- out1 = wasm_u32x4_extend_high_u16x8( l ); \
2016- out2 = wasm_u32x4_extend_low_u16x8 ( h ); \
2017- out3 = wasm_u32x4_extend_high_u16x8( h ); \
2018- }
2019-
2020- #define stbir__simdi_expand_u8_to_1u32(out,ireg) \
2021- { \
2022- v128_t tmp = wasm_u16x8_extend_low_u8x16(ireg); \
2023- out = wasm_u32x4_extend_low_u16x8(tmp); \
2024- }
2025-
2026- #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
2027- { \
2028- out0 = wasm_u32x4_extend_low_u16x8 ( ireg ); \
2029- out1 = wasm_u32x4_extend_high_u16x8( ireg ); \
2030- }
2031-
2032- #define stbir__simdf_convert_float_to_i32( i, f ) (i) = wasm_i32x4_trunc_sat_f32x4(f)
2033- #define stbir__simdf_convert_float_to_int( f ) wasm_i32x4_extract_lane(wasm_i32x4_trunc_sat_f32x4(f), 0)
2034- #define stbir__simdi_to_int( i ) wasm_i32x4_extract_lane(i, 0)
2035- #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)wasm_i32x4_extract_lane(wasm_i32x4_trunc_sat_f32x4(wasm_f32x4_max(wasm_f32x4_min(f,STBIR_max_uint8_as_float),wasm_f32x4_const_splat(0))), 0))
2036- #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)wasm_i32x4_extract_lane(wasm_i32x4_trunc_sat_f32x4(wasm_f32x4_max(wasm_f32x4_min(f,STBIR_max_uint16_as_float),wasm_f32x4_const_splat(0))), 0))
2037- #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = wasm_f32x4_convert_i32x4(ireg)
2038- #define stbir__simdf_add( out, reg0, reg1 ) (out) = wasm_f32x4_add( reg0, reg1 )
2039- #define stbir__simdf_mult( out, reg0, reg1 ) (out) = wasm_f32x4_mul( reg0, reg1 )
2040- #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = wasm_f32x4_mul( reg, wasm_v128_load( (void const*)(ptr) ) )
2041- #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = wasm_f32x4_mul( reg, wasm_v128_load32_splat( (void const*)(ptr) ) )
2042- #define stbir__simdf_add_mem( out, reg, ptr ) (out) = wasm_f32x4_add( reg, wasm_v128_load( (void const*)(ptr) ) )
2043- #define stbir__simdf_add1_mem( out, reg, ptr ) (out) = wasm_f32x4_add( reg, wasm_v128_load32_splat( (void const*)(ptr) ) )
2044-
2045- #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul1, mul2 ) )
2046- #define stbir__simdf_madd1( out, add, mul1, mul2 ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul1, mul2 ) )
2047- #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul, wasm_v128_load( (void const*)(ptr) ) ) )
2048- #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul, wasm_v128_load32_splat( (void const*)(ptr) ) ) )
2049-
2050- #define stbir__simdf_add1( out, reg0, reg1 ) (out) = wasm_f32x4_add( reg0, reg1 )
2051- #define stbir__simdf_mult1( out, reg0, reg1 ) (out) = wasm_f32x4_mul( reg0, reg1 )
2052-
2053- #define stbir__simdf_and( out, reg0, reg1 ) (out) = wasm_v128_and( reg0, reg1 )
2054- #define stbir__simdf_or( out, reg0, reg1 ) (out) = wasm_v128_or( reg0, reg1 )
2055-
2056- #define stbir__simdf_min( out, reg0, reg1 ) (out) = wasm_f32x4_min( reg0, reg1 )
2057- #define stbir__simdf_max( out, reg0, reg1 ) (out) = wasm_f32x4_max( reg0, reg1 )
2058- #define stbir__simdf_min1( out, reg0, reg1 ) (out) = wasm_f32x4_min( reg0, reg1 )
2059- #define stbir__simdf_max1( out, reg0, reg1 ) (out) = wasm_f32x4_max( reg0, reg1 )
2060-
2061- #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out) = wasm_i32x4_shuffle( reg0, reg1, 3, 4, 5, -1 )
2062- #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out) = wasm_i32x4_shuffle( reg0, reg1, 2, 3, 4, -1 )
2063-
2064- #define stbir__simdf_aaa1(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 3, 3, 3, 4)
2065- #define stbir__simdf_1aaa(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 4, 0, 0, 0)
2066- #define stbir__simdf_a1a1(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 1, 4, 3, 4)
2067- #define stbir__simdf_1a1a(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 4, 0, 4, 2)
2068-
2069- #define stbir__simdf_swiz( reg, one, two, three, four ) wasm_i32x4_shuffle(reg, reg, one, two, three, four)
2070-
2071- #define stbir__simdi_and( out, reg0, reg1 ) (out) = wasm_v128_and( reg0, reg1 )
2072- #define stbir__simdi_or( out, reg0, reg1 ) (out) = wasm_v128_or( reg0, reg1 )
2073- #define stbir__simdi_16madd( out, reg0, reg1 ) (out) = wasm_i32x4_dot_i16x8( reg0, reg1 )
2074-
2075- #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
2076- { \
2077- v128_t af = wasm_f32x4_max( wasm_f32x4_min(aa, STBIR_max_uint8_as_float), wasm_f32x4_const_splat(0) ); \
2078- v128_t bf = wasm_f32x4_max( wasm_f32x4_min(bb, STBIR_max_uint8_as_float), wasm_f32x4_const_splat(0) ); \
2079- v128_t ai = wasm_i32x4_trunc_sat_f32x4( af ); \
2080- v128_t bi = wasm_i32x4_trunc_sat_f32x4( bf ); \
2081- v128_t out16 = wasm_i16x8_narrow_i32x4( ai, bi ); \
2082- out = wasm_u8x16_narrow_i16x8( out16, out16 ); \
2083- }
2084-
2085- #define stbir__simdf_pack_to_8words(out,aa,bb) \
2086- { \
2087- v128_t af = wasm_f32x4_max( wasm_f32x4_min(aa, STBIR_max_uint16_as_float), wasm_f32x4_const_splat(0)); \
2088- v128_t bf = wasm_f32x4_max( wasm_f32x4_min(bb, STBIR_max_uint16_as_float), wasm_f32x4_const_splat(0)); \
2089- v128_t ai = wasm_i32x4_trunc_sat_f32x4( af ); \
2090- v128_t bi = wasm_i32x4_trunc_sat_f32x4( bf ); \
2091- out = wasm_u16x8_narrow_i32x4( ai, bi ); \
2092- }
2093-
2094- #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
2095- { \
2096- v128_t tmp0 = wasm_i16x8_narrow_i32x4(r0, r1); \
2097- v128_t tmp1 = wasm_i16x8_narrow_i32x4(r2, r3); \
2098- v128_t tmp = wasm_u8x16_narrow_i16x8(tmp0, tmp1); \
2099- tmp = wasm_i8x16_shuffle(tmp, tmp, 0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15); \
2100- wasm_v128_store( (void*)(ptr), tmp); \
2101- }
2102-
2103- #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
2104- { \
2105- v128_t t0 = wasm_v128_load( ptr ); \
2106- v128_t t1 = wasm_v128_load( ptr+4 ); \
2107- v128_t t2 = wasm_v128_load( ptr+8 ); \
2108- v128_t t3 = wasm_v128_load( ptr+12 ); \
2109- v128_t s0 = wasm_i32x4_shuffle(t0, t1, 0, 4, 2, 6); \
2110- v128_t s1 = wasm_i32x4_shuffle(t0, t1, 1, 5, 3, 7); \
2111- v128_t s2 = wasm_i32x4_shuffle(t2, t3, 0, 4, 2, 6); \
2112- v128_t s3 = wasm_i32x4_shuffle(t2, t3, 1, 5, 3, 7); \
2113- o0 = wasm_i32x4_shuffle(s0, s2, 0, 1, 4, 5); \
2114- o1 = wasm_i32x4_shuffle(s1, s3, 0, 1, 4, 5); \
2115- o2 = wasm_i32x4_shuffle(s0, s2, 2, 3, 6, 7); \
2116- o3 = wasm_i32x4_shuffle(s1, s3, 2, 3, 6, 7); \
2117- }
2118-
2119- #define stbir__simdi_32shr( out, reg, imm ) out = wasm_u32x4_shr( reg, imm )
2120-
2121- typedef float stbir__f32x4 __attribute__((__vector_size__(16), __aligned__(16)));
2122- #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = (v128_t)(stbir__f32x4){ x, x, x, x }
2123- #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { x, x, x, x }
2124- #define STBIR__CONSTF(var) (var)
2125- #define STBIR__CONSTI(var) (var)
2126-
2127- #ifdef STBIR_FLOORF
2128- #undef STBIR_FLOORF
2129- #endif
2130- #define STBIR_FLOORF stbir_simd_floorf
2131- static stbir__inline float stbir_simd_floorf(float x)
2132- {
2133- return wasm_f32x4_extract_lane( wasm_f32x4_floor( wasm_f32x4_splat(x) ), 0);
2134- }
2135-
2136- #ifdef STBIR_CEILF
2137- #undef STBIR_CEILF
2138- #endif
2139- #define STBIR_CEILF stbir_simd_ceilf
2140- static stbir__inline float stbir_simd_ceilf(float x)
2141- {
2142- return wasm_f32x4_extract_lane( wasm_f32x4_ceil( wasm_f32x4_splat(x) ), 0);
2143- }
2144-
2145- #define STBIR_SIMD
2146-
2147-#endif // SSE2/NEON/WASM
2148-
2149-#endif // NO SIMD
2150-
2151-#ifdef STBIR_SIMD8
2152- #define stbir__simdfX stbir__simdf8
2153- #define stbir__simdiX stbir__simdi8
2154- #define stbir__simdfX_load stbir__simdf8_load
2155- #define stbir__simdiX_load stbir__simdi8_load
2156- #define stbir__simdfX_mult stbir__simdf8_mult
2157- #define stbir__simdfX_add_mem stbir__simdf8_add_mem
2158- #define stbir__simdfX_madd_mem stbir__simdf8_madd_mem
2159- #define stbir__simdfX_store stbir__simdf8_store
2160- #define stbir__simdiX_store stbir__simdi8_store
2161- #define stbir__simdf_frepX stbir__simdf8_frep8
2162- #define stbir__simdfX_madd stbir__simdf8_madd
2163- #define stbir__simdfX_min stbir__simdf8_min
2164- #define stbir__simdfX_max stbir__simdf8_max
2165- #define stbir__simdfX_aaa1 stbir__simdf8_aaa1
2166- #define stbir__simdfX_1aaa stbir__simdf8_1aaa
2167- #define stbir__simdfX_a1a1 stbir__simdf8_a1a1
2168- #define stbir__simdfX_1a1a stbir__simdf8_1a1a
2169- #define stbir__simdfX_convert_float_to_i32 stbir__simdf8_convert_float_to_i32
2170- #define stbir__simdfX_pack_to_words stbir__simdf8_pack_to_16words
2171- #define stbir__simdfX_zero stbir__simdf8_zero
2172- #define STBIR_onesX STBIR_ones8
2173- #define STBIR_max_uint8_as_floatX STBIR_max_uint8_as_float8
2174- #define STBIR_max_uint16_as_floatX STBIR_max_uint16_as_float8
2175- #define STBIR_simd_point5X STBIR_simd_point58
2176- #define stbir__simdfX_float_count 8
2177- #define stbir__simdfX_0123to1230 stbir__simdf8_0123to12301230
2178- #define stbir__simdfX_0123to2103 stbir__simdf8_0123to21032103
2179- static const stbir__simdf8 STBIR_max_uint16_as_float_inverted8 = { stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted };
2180- static const stbir__simdf8 STBIR_max_uint8_as_float_inverted8 = { stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted };
2181- static const stbir__simdf8 STBIR_ones8 = { 1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0 };
2182- static const stbir__simdf8 STBIR_simd_point58 = { 0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5 };
2183- static const stbir__simdf8 STBIR_max_uint8_as_float8 = { stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float, stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float };
2184- static const stbir__simdf8 STBIR_max_uint16_as_float8 = { stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float, stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float };
2185-#else
2186- #define stbir__simdfX stbir__simdf
2187- #define stbir__simdiX stbir__simdi
2188- #define stbir__simdfX_load stbir__simdf_load
2189- #define stbir__simdiX_load stbir__simdi_load
2190- #define stbir__simdfX_mult stbir__simdf_mult
2191- #define stbir__simdfX_add_mem stbir__simdf_add_mem
2192- #define stbir__simdfX_madd_mem stbir__simdf_madd_mem
2193- #define stbir__simdfX_store stbir__simdf_store
2194- #define stbir__simdiX_store stbir__simdi_store
2195- #define stbir__simdf_frepX stbir__simdf_frep4
2196- #define stbir__simdfX_madd stbir__simdf_madd
2197- #define stbir__simdfX_min stbir__simdf_min
2198- #define stbir__simdfX_max stbir__simdf_max
2199- #define stbir__simdfX_aaa1 stbir__simdf_aaa1
2200- #define stbir__simdfX_1aaa stbir__simdf_1aaa
2201- #define stbir__simdfX_a1a1 stbir__simdf_a1a1
2202- #define stbir__simdfX_1a1a stbir__simdf_1a1a
2203- #define stbir__simdfX_convert_float_to_i32 stbir__simdf_convert_float_to_i32
2204- #define stbir__simdfX_pack_to_words stbir__simdf_pack_to_8words
2205- #define stbir__simdfX_zero stbir__simdf_zero
2206- #define STBIR_onesX STBIR__CONSTF(STBIR_ones)
2207- #define STBIR_simd_point5X STBIR__CONSTF(STBIR_simd_point5)
2208- #define STBIR_max_uint8_as_floatX STBIR__CONSTF(STBIR_max_uint8_as_float)
2209- #define STBIR_max_uint16_as_floatX STBIR__CONSTF(STBIR_max_uint16_as_float)
2210- #define stbir__simdfX_float_count 4
2211- #define stbir__if_simdf8_cast_to_simdf4( val ) ( val )
2212- #define stbir__simdfX_0123to1230 stbir__simdf_0123to1230
2213- #define stbir__simdfX_0123to2103 stbir__simdf_0123to2103
2214-#endif
2215-
2216-
2217-#if defined(STBIR_NEON) && !defined(_M_ARM) && !defined(__arm__)
2218-
2219- #if defined( _MSC_VER ) && !defined(__clang__)
2220- typedef __int16 stbir__FP16;
2221- #else
2222- typedef float16_t stbir__FP16;
2223- #endif
2224-
2225-#else // no NEON, or 32-bit ARM for MSVC
2226-
2227- typedef union stbir__FP16
2228- {
2229- unsigned short u;
2230- } stbir__FP16;
2231-
2232-#endif
2233-
2234-#if (!defined(STBIR_NEON) && !defined(STBIR_FP16C)) || (defined(STBIR_NEON) && defined(_M_ARM)) || (defined(STBIR_NEON) && defined(__arm__))
2235-
2236- // Fabian's half float routines, see: https://gist.github.com/rygorous/2156668
2237-
2238- static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2239- {
2240- static const stbir__FP32 magic = { (254 - 15) << 23 };
2241- static const stbir__FP32 was_infnan = { (127 + 16) << 23 };
2242- stbir__FP32 o;
2243-
2244- o.u = (h.u & 0x7fff) << 13; // exponent/mantissa bits
2245- o.f *= magic.f; // exponent adjust
2246- if (o.f >= was_infnan.f) // make sure Inf/NaN survive
2247- o.u |= 255 << 23;
2248- o.u |= (h.u & 0x8000) << 16; // sign bit
2249- return o.f;
2250- }
2251-
2252- static stbir__inline stbir__FP16 stbir__float_to_half(float val)
2253- {
2254- stbir__FP32 f32infty = { 255 << 23 };
2255- stbir__FP32 f16max = { (127 + 16) << 23 };
2256- stbir__FP32 denorm_magic = { ((127 - 15) + (23 - 10) + 1) << 23 };
2257- unsigned int sign_mask = 0x80000000u;
2258- stbir__FP16 o = { 0 };
2259- stbir__FP32 f;
2260- unsigned int sign;
2261-
2262- f.f = val;
2263- sign = f.u & sign_mask;
2264- f.u ^= sign;
2265-
2266- if (f.u >= f16max.u) // result is Inf or NaN (all exponent bits set)
2267- o.u = (f.u > f32infty.u) ? 0x7e00 : 0x7c00; // NaN->qNaN and Inf->Inf
2268- else // (De)normalized number or zero
2269- {
2270- if (f.u < (113 << 23)) // resulting FP16 is subnormal or zero
2271- {
2272- // use a magic value to align our 10 mantissa bits at the bottom of
2273- // the float. as long as FP addition is round-to-nearest-even this
2274- // just works.
2275- f.f += denorm_magic.f;
2276- // and one integer subtract of the bias later, we have our final float!
2277- o.u = (unsigned short) ( f.u - denorm_magic.u );
2278- }
2279- else
2280- {
2281- unsigned int mant_odd = (f.u >> 13) & 1; // resulting mantissa is odd
2282- // update exponent, rounding bias part 1
2283- f.u = f.u + ((15u - 127) << 23) + 0xfff;
2284- // rounding bias part 2
2285- f.u += mant_odd;
2286- // take the bits!
2287- o.u = (unsigned short) ( f.u >> 13 );
2288- }
2289- }
2290-
2291- o.u |= sign >> 16;
2292- return o;
2293- }
2294-
2295-#endif
2296-
2297-
2298-#if defined(STBIR_FP16C)
2299-
2300- #include <immintrin.h>
2301-
2302- static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2303- {
2304- _mm256_storeu_ps( (float*)output, _mm256_cvtph_ps( _mm_loadu_si128( (__m128i const* )input ) ) );
2305- }
2306-
2307- static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2308- {
2309- _mm_storeu_si128( (__m128i*)output, _mm256_cvtps_ph( _mm256_loadu_ps( input ), 0 ) );
2310- }
2311-
2312- static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2313- {
2314- return _mm_cvtss_f32( _mm_cvtph_ps( _mm_cvtsi32_si128( (int)h.u ) ) );
2315- }
2316-
2317- static stbir__inline stbir__FP16 stbir__float_to_half( float f )
2318- {
2319- stbir__FP16 h;
2320- h.u = (unsigned short) _mm_cvtsi128_si32( _mm_cvtps_ph( _mm_set_ss( f ), 0 ) );
2321- return h;
2322- }
2323-
2324-#elif defined(STBIR_SSE2)
2325-
2326- // Fabian's half float routines, see: https://gist.github.com/rygorous/2156668
2327- stbir__inline static void stbir__half_to_float_SIMD(float * output, void const * input)
2328- {
2329- static const STBIR__SIMDI_CONST(mask_nosign, 0x7fff);
2330- static const STBIR__SIMDI_CONST(smallest_normal, 0x0400);
2331- static const STBIR__SIMDI_CONST(infinity, 0x7c00);
2332- static const STBIR__SIMDI_CONST(expadjust_normal, (127 - 15) << 23);
2333- static const STBIR__SIMDI_CONST(magic_denorm, 113 << 23);
2334-
2335- __m128i i = _mm_loadu_si128 ( (__m128i const*)(input) );
2336- __m128i h = _mm_unpacklo_epi16 ( i, _mm_setzero_si128() );
2337- __m128i mnosign = STBIR__CONSTI(mask_nosign);
2338- __m128i eadjust = STBIR__CONSTI(expadjust_normal);
2339- __m128i smallest = STBIR__CONSTI(smallest_normal);
2340- __m128i infty = STBIR__CONSTI(infinity);
2341- __m128i expmant = _mm_and_si128(mnosign, h);
2342- __m128i justsign = _mm_xor_si128(h, expmant);
2343- __m128i b_notinfnan = _mm_cmpgt_epi32(infty, expmant);
2344- __m128i b_isdenorm = _mm_cmpgt_epi32(smallest, expmant);
2345- __m128i shifted = _mm_slli_epi32(expmant, 13);
2346- __m128i adj_infnan = _mm_andnot_si128(b_notinfnan, eadjust);
2347- __m128i adjusted = _mm_add_epi32(eadjust, shifted);
2348- __m128i den1 = _mm_add_epi32(shifted, STBIR__CONSTI(magic_denorm));
2349- __m128i adjusted2 = _mm_add_epi32(adjusted, adj_infnan);
2350- __m128 den2 = _mm_sub_ps(_mm_castsi128_ps(den1), *(const __m128 *)&magic_denorm);
2351- __m128 adjusted3 = _mm_and_ps(den2, _mm_castsi128_ps(b_isdenorm));
2352- __m128 adjusted4 = _mm_andnot_ps(_mm_castsi128_ps(b_isdenorm), _mm_castsi128_ps(adjusted2));
2353- __m128 adjusted5 = _mm_or_ps(adjusted3, adjusted4);
2354- __m128i sign = _mm_slli_epi32(justsign, 16);
2355- __m128 final = _mm_or_ps(adjusted5, _mm_castsi128_ps(sign));
2356- stbir__simdf_store( output + 0, final );
2357-
2358- h = _mm_unpackhi_epi16 ( i, _mm_setzero_si128() );
2359- expmant = _mm_and_si128(mnosign, h);
2360- justsign = _mm_xor_si128(h, expmant);
2361- b_notinfnan = _mm_cmpgt_epi32(infty, expmant);
2362- b_isdenorm = _mm_cmpgt_epi32(smallest, expmant);
2363- shifted = _mm_slli_epi32(expmant, 13);
2364- adj_infnan = _mm_andnot_si128(b_notinfnan, eadjust);
2365- adjusted = _mm_add_epi32(eadjust, shifted);
2366- den1 = _mm_add_epi32(shifted, STBIR__CONSTI(magic_denorm));
2367- adjusted2 = _mm_add_epi32(adjusted, adj_infnan);
2368- den2 = _mm_sub_ps(_mm_castsi128_ps(den1), *(const __m128 *)&magic_denorm);
2369- adjusted3 = _mm_and_ps(den2, _mm_castsi128_ps(b_isdenorm));
2370- adjusted4 = _mm_andnot_ps(_mm_castsi128_ps(b_isdenorm), _mm_castsi128_ps(adjusted2));
2371- adjusted5 = _mm_or_ps(adjusted3, adjusted4);
2372- sign = _mm_slli_epi32(justsign, 16);
2373- final = _mm_or_ps(adjusted5, _mm_castsi128_ps(sign));
2374- stbir__simdf_store( output + 4, final );
2375-
2376- // ~38 SSE2 ops for 8 values
2377- }
2378-
2379- // Fabian's round-to-nearest-even float to half
2380- // ~48 SSE2 ops for 8 output
2381- stbir__inline static void stbir__float_to_half_SIMD(void * output, float const * input)
2382- {
2383- static const STBIR__SIMDI_CONST(mask_sign, 0x80000000u);
2384- static const STBIR__SIMDI_CONST(c_f16max, (127 + 16) << 23); // all FP32 values >=this round to +inf
2385- static const STBIR__SIMDI_CONST(c_nanbit, 0x200);
2386- static const STBIR__SIMDI_CONST(c_infty_as_fp16, 0x7c00);
2387- static const STBIR__SIMDI_CONST(c_min_normal, (127 - 14) << 23); // smallest FP32 that yields a normalized FP16
2388- static const STBIR__SIMDI_CONST(c_subnorm_magic, ((127 - 15) + (23 - 10) + 1) << 23);
2389- static const STBIR__SIMDI_CONST(c_normal_bias, 0xfff - ((127 - 15) << 23)); // adjust exponent and add mantissa rounding
2390-
2391- __m128 f = _mm_loadu_ps(input);
2392- __m128 msign = _mm_castsi128_ps(STBIR__CONSTI(mask_sign));
2393- __m128 justsign = _mm_and_ps(msign, f);
2394- __m128 absf = _mm_xor_ps(f, justsign);
2395- __m128i absf_int = _mm_castps_si128(absf); // the cast is "free" (extra bypass latency, but no thruput hit)
2396- __m128i f16max = STBIR__CONSTI(c_f16max);
2397- __m128 b_isnan = _mm_cmpunord_ps(absf, absf); // is this a NaN?
2398- __m128i b_isregular = _mm_cmpgt_epi32(f16max, absf_int); // (sub)normalized or special?
2399- __m128i nanbit = _mm_and_si128(_mm_castps_si128(b_isnan), STBIR__CONSTI(c_nanbit));
2400- __m128i inf_or_nan = _mm_or_si128(nanbit, STBIR__CONSTI(c_infty_as_fp16)); // output for specials
2401-
2402- __m128i min_normal = STBIR__CONSTI(c_min_normal);
2403- __m128i b_issub = _mm_cmpgt_epi32(min_normal, absf_int);
2404-
2405- // "result is subnormal" path
2406- __m128 subnorm1 = _mm_add_ps(absf, _mm_castsi128_ps(STBIR__CONSTI(c_subnorm_magic))); // magic value to round output mantissa
2407- __m128i subnorm2 = _mm_sub_epi32(_mm_castps_si128(subnorm1), STBIR__CONSTI(c_subnorm_magic)); // subtract out bias
2408-
2409- // "result is normal" path
2410- __m128i mantoddbit = _mm_slli_epi32(absf_int, 31 - 13); // shift bit 13 (mantissa LSB) to sign
2411- __m128i mantodd = _mm_srai_epi32(mantoddbit, 31); // -1 if FP16 mantissa odd, else 0
2412-
2413- __m128i round1 = _mm_add_epi32(absf_int, STBIR__CONSTI(c_normal_bias));
2414- __m128i round2 = _mm_sub_epi32(round1, mantodd); // if mantissa LSB odd, bias towards rounding up (RTNE)
2415- __m128i normal = _mm_srli_epi32(round2, 13); // rounded result
2416-
2417- // combine the two non-specials
2418- __m128i nonspecial = _mm_or_si128(_mm_and_si128(subnorm2, b_issub), _mm_andnot_si128(b_issub, normal));
2419-
2420- // merge in specials as well
2421- __m128i joined = _mm_or_si128(_mm_and_si128(nonspecial, b_isregular), _mm_andnot_si128(b_isregular, inf_or_nan));
2422-
2423- __m128i sign_shift = _mm_srai_epi32(_mm_castps_si128(justsign), 16);
2424- __m128i final2, final= _mm_or_si128(joined, sign_shift);
2425-
2426- f = _mm_loadu_ps(input+4);
2427- justsign = _mm_and_ps(msign, f);
2428- absf = _mm_xor_ps(f, justsign);
2429- absf_int = _mm_castps_si128(absf); // the cast is "free" (extra bypass latency, but no thruput hit)
2430- b_isnan = _mm_cmpunord_ps(absf, absf); // is this a NaN?
2431- b_isregular = _mm_cmpgt_epi32(f16max, absf_int); // (sub)normalized or special?
2432- nanbit = _mm_and_si128(_mm_castps_si128(b_isnan), c_nanbit);
2433- inf_or_nan = _mm_or_si128(nanbit, STBIR__CONSTI(c_infty_as_fp16)); // output for specials
2434-
2435- b_issub = _mm_cmpgt_epi32(min_normal, absf_int);
2436-
2437- // "result is subnormal" path
2438- subnorm1 = _mm_add_ps(absf, _mm_castsi128_ps(STBIR__CONSTI(c_subnorm_magic))); // magic value to round output mantissa
2439- subnorm2 = _mm_sub_epi32(_mm_castps_si128(subnorm1), STBIR__CONSTI(c_subnorm_magic)); // subtract out bias
2440-
2441- // "result is normal" path
2442- mantoddbit = _mm_slli_epi32(absf_int, 31 - 13); // shift bit 13 (mantissa LSB) to sign
2443- mantodd = _mm_srai_epi32(mantoddbit, 31); // -1 if FP16 mantissa odd, else 0
2444-
2445- round1 = _mm_add_epi32(absf_int, STBIR__CONSTI(c_normal_bias));
2446- round2 = _mm_sub_epi32(round1, mantodd); // if mantissa LSB odd, bias towards rounding up (RTNE)
2447- normal = _mm_srli_epi32(round2, 13); // rounded result
2448-
2449- // combine the two non-specials
2450- nonspecial = _mm_or_si128(_mm_and_si128(subnorm2, b_issub), _mm_andnot_si128(b_issub, normal));
2451-
2452- // merge in specials as well
2453- joined = _mm_or_si128(_mm_and_si128(nonspecial, b_isregular), _mm_andnot_si128(b_isregular, inf_or_nan));
2454-
2455- sign_shift = _mm_srai_epi32(_mm_castps_si128(justsign), 16);
2456- final2 = _mm_or_si128(joined, sign_shift);
2457- final = _mm_packs_epi32(final, final2);
2458- stbir__simdi_store( output,final );
2459- }
2460-
2461-#elif defined(STBIR_NEON) && defined(_MSC_VER) && defined(_M_ARM64) && !defined(__clang__) // 64-bit ARM on MSVC (not clang)
2462-
2463- static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2464- {
2465- float16x4_t in0 = vld1_f16(input + 0);
2466- float16x4_t in1 = vld1_f16(input + 4);
2467- vst1q_f32(output + 0, vcvt_f32_f16(in0));
2468- vst1q_f32(output + 4, vcvt_f32_f16(in1));
2469- }
2470-
2471- static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2472- {
2473- float16x4_t out0 = vcvt_f16_f32(vld1q_f32(input + 0));
2474- float16x4_t out1 = vcvt_f16_f32(vld1q_f32(input + 4));
2475- vst1_f16(output+0, out0);
2476- vst1_f16(output+4, out1);
2477- }
2478-
2479- static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2480- {
2481- return vgetq_lane_f32(vcvt_f32_f16(vld1_dup_f16(&h)), 0);
2482- }
2483-
2484- static stbir__inline stbir__FP16 stbir__float_to_half( float f )
2485- {
2486- return vget_lane_f16(vcvt_f16_f32(vdupq_n_f32(f)), 0).n16_u16[0];
2487- }
2488-
2489-#elif defined(STBIR_NEON) && ( defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ ) ) // 64-bit ARM
2490-
2491- static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2492- {
2493- float16x8_t in = vld1q_f16(input);
2494- vst1q_f32(output + 0, vcvt_f32_f16(vget_low_f16(in)));
2495- vst1q_f32(output + 4, vcvt_f32_f16(vget_high_f16(in)));
2496- }
2497-
2498- static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2499- {
2500- float16x4_t out0 = vcvt_f16_f32(vld1q_f32(input + 0));
2501- float16x4_t out1 = vcvt_f16_f32(vld1q_f32(input + 4));
2502- vst1q_f16(output, vcombine_f16(out0, out1));
2503- }
2504-
2505- static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2506- {
2507- return vgetq_lane_f32(vcvt_f32_f16(vdup_n_f16(h)), 0);
2508- }
2509-
2510- static stbir__inline stbir__FP16 stbir__float_to_half( float f )
2511- {
2512- return vget_lane_f16(vcvt_f16_f32(vdupq_n_f32(f)), 0);
2513- }
2514-
2515-#elif defined(STBIR_WASM) || (defined(STBIR_NEON) && (defined(_MSC_VER) || defined(_M_ARM) || defined(__arm__))) // WASM or 32-bit ARM on MSVC/clang
2516-
2517- static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2518- {
2519- for (int i=0; i<8; i++)
2520- {
2521- output[i] = stbir__half_to_float(input[i]);
2522- }
2523- }
2524- static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2525- {
2526- for (int i=0; i<8; i++)
2527- {
2528- output[i] = stbir__float_to_half(input[i]);
2529- }
2530- }
2531-
2532-#endif
2533-
2534-
2535-#ifdef STBIR_SIMD
2536-
2537-#define stbir__simdf_0123to3333( out, reg ) (out) = stbir__simdf_swiz( reg, 3,3,3,3 )
2538-#define stbir__simdf_0123to2222( out, reg ) (out) = stbir__simdf_swiz( reg, 2,2,2,2 )
2539-#define stbir__simdf_0123to1111( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,1,1 )
2540-#define stbir__simdf_0123to0000( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,0,0 )
2541-#define stbir__simdf_0123to0003( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,0,3 )
2542-#define stbir__simdf_0123to0001( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,0,1 )
2543-#define stbir__simdf_0123to1122( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,2,2 )
2544-#define stbir__simdf_0123to2333( out, reg ) (out) = stbir__simdf_swiz( reg, 2,3,3,3 )
2545-#define stbir__simdf_0123to0023( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,2,3 )
2546-#define stbir__simdf_0123to1230( out, reg ) (out) = stbir__simdf_swiz( reg, 1,2,3,0 )
2547-#define stbir__simdf_0123to2103( out, reg ) (out) = stbir__simdf_swiz( reg, 2,1,0,3 )
2548-#define stbir__simdf_0123to3210( out, reg ) (out) = stbir__simdf_swiz( reg, 3,2,1,0 )
2549-#define stbir__simdf_0123to2301( out, reg ) (out) = stbir__simdf_swiz( reg, 2,3,0,1 )
2550-#define stbir__simdf_0123to3012( out, reg ) (out) = stbir__simdf_swiz( reg, 3,0,1,2 )
2551-#define stbir__simdf_0123to0011( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,1,1 )
2552-#define stbir__simdf_0123to1100( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,0,0 )
2553-#define stbir__simdf_0123to2233( out, reg ) (out) = stbir__simdf_swiz( reg, 2,2,3,3 )
2554-#define stbir__simdf_0123to1133( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,3,3 )
2555-#define stbir__simdf_0123to0022( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,2,2 )
2556-#define stbir__simdf_0123to1032( out, reg ) (out) = stbir__simdf_swiz( reg, 1,0,3,2 )
2557-
2558-typedef union stbir__simdi_u32
2559-{
2560- stbir_uint32 m128i_u32[4];
2561- int m128i_i32[4];
2562- stbir__simdi m128i_i128;
2563-} stbir__simdi_u32;
2564-
2565-static const int STBIR_mask[9] = { 0,0,0,-1,-1,-1,0,0,0 };
2566-
2567-static const STBIR__SIMDF_CONST(STBIR_max_uint8_as_float, stbir__max_uint8_as_float);
2568-static const STBIR__SIMDF_CONST(STBIR_max_uint16_as_float, stbir__max_uint16_as_float);
2569-static const STBIR__SIMDF_CONST(STBIR_max_uint8_as_float_inverted, stbir__max_uint8_as_float_inverted);
2570-static const STBIR__SIMDF_CONST(STBIR_max_uint16_as_float_inverted, stbir__max_uint16_as_float_inverted);
2571-
2572-static const STBIR__SIMDF_CONST(STBIR_simd_point5, 0.5f);
2573-static const STBIR__SIMDF_CONST(STBIR_ones, 1.0f);
2574-static const STBIR__SIMDI_CONST(STBIR_almost_zero, (127 - 13) << 23);
2575-static const STBIR__SIMDI_CONST(STBIR_almost_one, 0x3f7fffff);
2576-static const STBIR__SIMDI_CONST(STBIR_mantissa_mask, 0xff);
2577-static const STBIR__SIMDI_CONST(STBIR_topscale, 0x02000000);
2578-
2579-// Basically, in simd mode, we unroll the proper amount, and we don't want
2580-// the non-simd remnant loops to be unroll because they only run a few times
2581-// Adding this switch saves about 5K on clang which is Captain Unroll the 3rd.
2582-#define STBIR_SIMD_STREAMOUT_PTR( star ) STBIR_STREAMOUT_PTR( star )
2583-#define STBIR_SIMD_NO_UNROLL(ptr) STBIR_NO_UNROLL(ptr)
2584-#define STBIR_SIMD_NO_UNROLL_LOOP_START STBIR_NO_UNROLL_LOOP_START
2585-#define STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR STBIR_NO_UNROLL_LOOP_START_INF_FOR
2586-
2587-#ifdef STBIR_MEMCPY
2588-#undef STBIR_MEMCPY
2589-#endif
2590-#define STBIR_MEMCPY stbir_simd_memcpy
2591-
2592-// override normal use of memcpy with much simpler copy (faster and smaller with our sized copies)
2593-static void stbir_simd_memcpy( void * dest, void const * src, size_t bytes )
2594-{
2595- char STBIR_SIMD_STREAMOUT_PTR (*) d = (char*) dest;
2596- char STBIR_SIMD_STREAMOUT_PTR( * ) d_end = ((char*) dest) + bytes;
2597- ptrdiff_t ofs_to_src = (char*)src - (char*)dest;
2598-
2599- // check overlaps
2600- STBIR_ASSERT( ( ( d >= ( (char*)src) + bytes ) ) || ( ( d + bytes ) <= (char*)src ) );
2601-
2602- if ( bytes < (16*stbir__simdfX_float_count) )
2603- {
2604- if ( bytes < 16 )
2605- {
2606- if ( bytes )
2607- {
2608- STBIR_SIMD_NO_UNROLL_LOOP_START
2609- do
2610- {
2611- STBIR_SIMD_NO_UNROLL(d);
2612- d[ 0 ] = d[ ofs_to_src ];
2613- ++d;
2614- } while ( d < d_end );
2615- }
2616- }
2617- else
2618- {
2619- stbir__simdf x;
2620- // do one unaligned to get us aligned for the stream out below
2621- stbir__simdf_load( x, ( d + ofs_to_src ) );
2622- stbir__simdf_store( d, x );
2623- d = (char*)( ( ( (size_t)d ) + 16 ) & ~15 );
2624-
2625- STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
2626- for(;;)
2627- {
2628- STBIR_SIMD_NO_UNROLL(d);
2629-
2630- if ( d > ( d_end - 16 ) )
2631- {
2632- if ( d == d_end )
2633- return;
2634- d = d_end - 16;
2635- }
2636-
2637- stbir__simdf_load( x, ( d + ofs_to_src ) );
2638- stbir__simdf_store( d, x );
2639- d += 16;
2640- }
2641- }
2642- }
2643- else
2644- {
2645- stbir__simdfX x0,x1,x2,x3;
2646-
2647- // do one unaligned to get us aligned for the stream out below
2648- stbir__simdfX_load( x0, ( d + ofs_to_src ) + 0*stbir__simdfX_float_count );
2649- stbir__simdfX_load( x1, ( d + ofs_to_src ) + 4*stbir__simdfX_float_count );
2650- stbir__simdfX_load( x2, ( d + ofs_to_src ) + 8*stbir__simdfX_float_count );
2651- stbir__simdfX_load( x3, ( d + ofs_to_src ) + 12*stbir__simdfX_float_count );
2652- stbir__simdfX_store( d + 0*stbir__simdfX_float_count, x0 );
2653- stbir__simdfX_store( d + 4*stbir__simdfX_float_count, x1 );
2654- stbir__simdfX_store( d + 8*stbir__simdfX_float_count, x2 );
2655- stbir__simdfX_store( d + 12*stbir__simdfX_float_count, x3 );
2656- d = (char*)( ( ( (size_t)d ) + (16*stbir__simdfX_float_count) ) & ~((16*stbir__simdfX_float_count)-1) );
2657-
2658- STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
2659- for(;;)
2660- {
2661- STBIR_SIMD_NO_UNROLL(d);
2662-
2663- if ( d > ( d_end - (16*stbir__simdfX_float_count) ) )
2664- {
2665- if ( d == d_end )
2666- return;
2667- d = d_end - (16*stbir__simdfX_float_count);
2668- }
2669-
2670- stbir__simdfX_load( x0, ( d + ofs_to_src ) + 0*stbir__simdfX_float_count );
2671- stbir__simdfX_load( x1, ( d + ofs_to_src ) + 4*stbir__simdfX_float_count );
2672- stbir__simdfX_load( x2, ( d + ofs_to_src ) + 8*stbir__simdfX_float_count );
2673- stbir__simdfX_load( x3, ( d + ofs_to_src ) + 12*stbir__simdfX_float_count );
2674- stbir__simdfX_store( d + 0*stbir__simdfX_float_count, x0 );
2675- stbir__simdfX_store( d + 4*stbir__simdfX_float_count, x1 );
2676- stbir__simdfX_store( d + 8*stbir__simdfX_float_count, x2 );
2677- stbir__simdfX_store( d + 12*stbir__simdfX_float_count, x3 );
2678- d += (16*stbir__simdfX_float_count);
2679- }
2680- }
2681-}
2682-
2683-// memcpy that is specically intentionally overlapping (src is smaller then dest, so can be
2684-// a normal forward copy, bytes is divisible by 4 and bytes is greater than or equal to
2685-// the diff between dest and src)
2686-static void stbir_overlapping_memcpy( void * dest, void const * src, size_t bytes )
2687-{
2688- char STBIR_SIMD_STREAMOUT_PTR (*) sd = (char*) src;
2689- char STBIR_SIMD_STREAMOUT_PTR( * ) s_end = ((char*) src) + bytes;
2690- ptrdiff_t ofs_to_dest = (char*)dest - (char*)src;
2691-
2692- if ( ofs_to_dest >= 16 ) // is the overlap more than 16 away?
2693- {
2694- char STBIR_SIMD_STREAMOUT_PTR( * ) s_end16 = ((char*) src) + (bytes&~15);
2695- STBIR_SIMD_NO_UNROLL_LOOP_START
2696- do
2697- {
2698- stbir__simdf x;
2699- STBIR_SIMD_NO_UNROLL(sd);
2700- stbir__simdf_load( x, sd );
2701- stbir__simdf_store( ( sd + ofs_to_dest ), x );
2702- sd += 16;
2703- } while ( sd < s_end16 );
2704-
2705- if ( sd == s_end )
2706- return;
2707- }
2708-
2709- do
2710- {
2711- STBIR_SIMD_NO_UNROLL(sd);
2712- *(int*)( sd + ofs_to_dest ) = *(int*) sd;
2713- sd += 4;
2714- } while ( sd < s_end );
2715-}
2716-
2717-#else // no SSE2
2718-
2719-// when in scalar mode, we let unrolling happen, so this macro just does the __restrict
2720-#define STBIR_SIMD_STREAMOUT_PTR( star ) STBIR_STREAMOUT_PTR( star )
2721-#define STBIR_SIMD_NO_UNROLL(ptr)
2722-#define STBIR_SIMD_NO_UNROLL_LOOP_START
2723-#define STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
2724-
2725-#endif // SSE2
2726-
2727-
2728-#ifdef STBIR_PROFILE
2729-
2730-#ifndef STBIR_PROFILE_FUNC
2731-
2732-#if defined(_x86_64) || defined( __x86_64__ ) || defined( _M_X64 ) || defined(__x86_64) || defined(__SSE2__) || defined(STBIR_SSE) || defined( _M_IX86_FP ) || defined(__i386) || defined( __i386__ ) || defined( _M_IX86 ) || defined( _X86_ )
2733-
2734-#ifdef _MSC_VER
2735-
2736- STBIRDEF stbir_uint64 __rdtsc();
2737- #define STBIR_PROFILE_FUNC() __rdtsc()
2738-
2739-#else // non msvc
2740-
2741- static stbir__inline stbir_uint64 STBIR_PROFILE_FUNC()
2742- {
2743- stbir_uint32 lo, hi;
2744- asm volatile ("rdtsc" : "=a" (lo), "=d" (hi) );
2745- return ( ( (stbir_uint64) hi ) << 32 ) | ( (stbir_uint64) lo );
2746- }
2747-
2748-#endif // msvc
2749-
2750-#elif defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ ) || defined(__ARM_NEON__)
2751-
2752-#if defined( _MSC_VER ) && !defined(__clang__)
2753-
2754- #define STBIR_PROFILE_FUNC() _ReadStatusReg(ARM64_CNTVCT)
2755-
2756-#else
2757-
2758- static stbir__inline stbir_uint64 STBIR_PROFILE_FUNC()
2759- {
2760- stbir_uint64 tsc;
2761- asm volatile("mrs %0, cntvct_el0" : "=r" (tsc));
2762- return tsc;
2763- }
2764-
2765-#endif
2766-
2767-#else // x64, arm
2768-
2769-#error Unknown platform for profiling.
2770-
2771-#endif // x64, arm
2772-
2773-#endif // STBIR_PROFILE_FUNC
2774-
2775-#define STBIR_ONLY_PROFILE_GET_SPLIT_INFO ,stbir__per_split_info * split_info
2776-#define STBIR_ONLY_PROFILE_SET_SPLIT_INFO ,split_info
2777-
2778-#define STBIR_ONLY_PROFILE_BUILD_GET_INFO ,stbir__info * profile_info
2779-#define STBIR_ONLY_PROFILE_BUILD_SET_INFO ,profile_info
2780-
2781-// super light-weight micro profiler
2782-#define STBIR_PROFILE_START_ll( info, wh ) { stbir_uint64 wh##thiszonetime = STBIR_PROFILE_FUNC(); stbir_uint64 * wh##save_parent_excluded_ptr = info->current_zone_excluded_ptr; stbir_uint64 wh##current_zone_excluded = 0; info->current_zone_excluded_ptr = &wh##current_zone_excluded;
2783-#define STBIR_PROFILE_END_ll( info, wh ) wh##thiszonetime = STBIR_PROFILE_FUNC() - wh##thiszonetime; info->profile.named.wh += wh##thiszonetime - wh##current_zone_excluded; *wh##save_parent_excluded_ptr += wh##thiszonetime; info->current_zone_excluded_ptr = wh##save_parent_excluded_ptr; }
2784-#define STBIR_PROFILE_FIRST_START_ll( info, wh ) { int i; info->current_zone_excluded_ptr = &info->profile.named.total; for(i=0;i<STBIR__ARRAY_SIZE(info->profile.array);i++) info->profile.array[i]=0; } STBIR_PROFILE_START_ll( info, wh );
2785-#define STBIR_PROFILE_CLEAR_EXTRAS_ll( info, num ) { int extra; for(extra=1;extra<(num);extra++) { int i; for(i=0;i<STBIR__ARRAY_SIZE((info)->profile.array);i++) (info)[extra].profile.array[i]=0; } }
2786-
2787-// for thread data
2788-#define STBIR_PROFILE_START( wh ) STBIR_PROFILE_START_ll( split_info, wh )
2789-#define STBIR_PROFILE_END( wh ) STBIR_PROFILE_END_ll( split_info, wh )
2790-#define STBIR_PROFILE_FIRST_START( wh ) STBIR_PROFILE_FIRST_START_ll( split_info, wh )
2791-#define STBIR_PROFILE_CLEAR_EXTRAS() STBIR_PROFILE_CLEAR_EXTRAS_ll( split_info, split_count )
2792-
2793-// for build data
2794-#define STBIR_PROFILE_BUILD_START( wh ) STBIR_PROFILE_START_ll( profile_info, wh )
2795-#define STBIR_PROFILE_BUILD_END( wh ) STBIR_PROFILE_END_ll( profile_info, wh )
2796-#define STBIR_PROFILE_BUILD_FIRST_START( wh ) STBIR_PROFILE_FIRST_START_ll( profile_info, wh )
2797-#define STBIR_PROFILE_BUILD_CLEAR( info ) { int i; for(i=0;i<STBIR__ARRAY_SIZE(info->profile.array);i++) info->profile.array[i]=0; }
2798-
2799-#else // no profile
2800-
2801-#define STBIR_ONLY_PROFILE_GET_SPLIT_INFO
2802-#define STBIR_ONLY_PROFILE_SET_SPLIT_INFO
2803-
2804-#define STBIR_ONLY_PROFILE_BUILD_GET_INFO
2805-#define STBIR_ONLY_PROFILE_BUILD_SET_INFO
2806-
2807-#define STBIR_PROFILE_START( wh )
2808-#define STBIR_PROFILE_END( wh )
2809-#define STBIR_PROFILE_FIRST_START( wh )
2810-#define STBIR_PROFILE_CLEAR_EXTRAS( )
2811-
2812-#define STBIR_PROFILE_BUILD_START( wh )
2813-#define STBIR_PROFILE_BUILD_END( wh )
2814-#define STBIR_PROFILE_BUILD_FIRST_START( wh )
2815-#define STBIR_PROFILE_BUILD_CLEAR( info )
2816-
2817-#endif // stbir_profile
2818-
2819-#ifndef STBIR_CEILF
2820-#include <math.h>
2821-#if _MSC_VER <= 1200 // support VC6 for Sean
2822-#define STBIR_CEILF(x) ((float)ceil((float)(x)))
2823-#define STBIR_FLOORF(x) ((float)floor((float)(x)))
2824-#else
2825-#define STBIR_CEILF(x) ceilf(x)
2826-#define STBIR_FLOORF(x) floorf(x)
2827-#endif
2828-#endif
2829-
2830-#ifndef STBIR_MEMCPY
2831-// For memcpy
2832-#include <string.h>
2833-#define STBIR_MEMCPY( dest, src, len ) memcpy( dest, src, len )
2834-#endif
2835-
2836-#ifndef STBIR_SIMD
2837-
2838-// memcpy that is specifically intentionally overlapping (src is smaller then dest, so can be
2839-// a normal forward copy, bytes is divisible by 4 and bytes is greater than or equal to
2840-// the diff between dest and src)
2841-static void stbir_overlapping_memcpy( void * dest, void const * src, size_t bytes )
2842-{
2843- char STBIR_SIMD_STREAMOUT_PTR (*) sd = (char*) src;
2844- char STBIR_SIMD_STREAMOUT_PTR( * ) s_end = ((char*) src) + bytes;
2845- ptrdiff_t ofs_to_dest = (char*)dest - (char*)src;
2846-
2847- if ( ofs_to_dest >= 8 ) // is the overlap more than 8 away
2848- {
2849- char STBIR_SIMD_STREAMOUT_PTR( * ) s_end8 = ((char*) src) + (bytes&~7);
2850-
2851- if ( ( ( ((ptrdiff_t)dest)|((ptrdiff_t)src) ) & 7 ) == 0 ) // is it 8byte aligned?
2852- {
2853- STBIR_NO_UNROLL_LOOP_START
2854- do
2855- {
2856- STBIR_NO_UNROLL(sd);
2857- *(stbir_uint64*)( sd + ofs_to_dest ) = *(stbir_uint64*) sd;
2858- sd += 8;
2859- } while ( sd < s_end8 );
2860- }
2861- else
2862- {
2863- STBIR_NO_UNROLL_LOOP_START
2864- do
2865- {
2866- int a,b;
2867- STBIR_NO_UNROLL(sd);
2868- a = ((int*)sd)[0];
2869- b = ((int*)sd)[1];
2870- ((int*)( sd + ofs_to_dest ))[0] = a;
2871- ((int*)( sd + ofs_to_dest ))[1] = b;
2872- sd += 8;
2873- } while ( sd < s_end8 );
2874- }
2875-
2876- if ( sd == s_end )
2877- return;
2878- }
2879-
2880- STBIR_NO_UNROLL_LOOP_START
2881- do
2882- {
2883- STBIR_NO_UNROLL(sd);
2884- *(int*)( sd + ofs_to_dest ) = *(int*) sd;
2885- sd += 4;
2886- } while ( sd < s_end );
2887-}
2888-
2889-#endif
2890-
2891-static float stbir__filter_trapezoid(float x, float scale, void * user_data)
2892-{
2893- float halfscale = scale / 2;
2894- float t = 0.5f + halfscale;
2895- STBIR_ASSERT(scale <= 1);
2896- STBIR__UNUSED(user_data);
2897-
2898- if ( x < 0.0f ) x = -x;
2899-
2900- if (x >= t)
2901- return 0.0f;
2902- else
2903- {
2904- float r = 0.5f - halfscale;
2905- if (x <= r)
2906- return 1.0f;
2907- else
2908- return (t - x) / scale;
2909- }
2910-}
2911-
2912-static float stbir__support_trapezoid(float scale, void * user_data)
2913-{
2914- STBIR__UNUSED(user_data);
2915- return 0.5f + scale / 2.0f;
2916-}
2917-
2918-static float stbir__filter_triangle(float x, float s, void * user_data)
2919-{
2920- STBIR__UNUSED(s);
2921- STBIR__UNUSED(user_data);
2922-
2923- if ( x < 0.0f ) x = -x;
2924-
2925- if (x <= 1.0f)
2926- return 1.0f - x;
2927- else
2928- return 0.0f;
2929-}
2930-
2931-static float stbir__filter_point(float x, float s, void * user_data)
2932-{
2933- STBIR__UNUSED(x);
2934- STBIR__UNUSED(s);
2935- STBIR__UNUSED(user_data);
2936-
2937- return 1.0f;
2938-}
2939-
2940-static float stbir__filter_cubic(float x, float s, void * user_data)
2941-{
2942- STBIR__UNUSED(s);
2943- STBIR__UNUSED(user_data);
2944-
2945- if ( x < 0.0f ) x = -x;
2946-
2947- if (x < 1.0f)
2948- return (4.0f + x*x*(3.0f*x - 6.0f))/6.0f;
2949- else if (x < 2.0f)
2950- return (8.0f + x*(-12.0f + x*(6.0f - x)))/6.0f;
2951-
2952- return (0.0f);
2953-}
2954-
2955-static float stbir__filter_catmullrom(float x, float s, void * user_data)
2956-{
2957- STBIR__UNUSED(s);
2958- STBIR__UNUSED(user_data);
2959-
2960- if ( x < 0.0f ) x = -x;
2961-
2962- if (x < 1.0f)
2963- return 1.0f - x*x*(2.5f - 1.5f*x);
2964- else if (x < 2.0f)
2965- return 2.0f - x*(4.0f + x*(0.5f*x - 2.5f));
2966-
2967- return (0.0f);
2968-}
2969-
2970-static float stbir__filter_mitchell(float x, float s, void * user_data)
2971-{
2972- STBIR__UNUSED(s);
2973- STBIR__UNUSED(user_data);
2974-
2975- if ( x < 0.0f ) x = -x;
2976-
2977- if (x < 1.0f)
2978- return (16.0f + x*x*(21.0f * x - 36.0f))/18.0f;
2979- else if (x < 2.0f)
2980- return (32.0f + x*(-60.0f + x*(36.0f - 7.0f*x)))/18.0f;
2981-
2982- return (0.0f);
2983-}
2984-
2985-static float stbir__support_zeropoint5(float s, void * user_data)
2986-{
2987- STBIR__UNUSED(s);
2988- STBIR__UNUSED(user_data);
2989- return 0.5f;
2990-}
2991-
2992-static float stbir__support_one(float s, void * user_data)
2993-{
2994- STBIR__UNUSED(s);
2995- STBIR__UNUSED(user_data);
2996- return 1;
2997-}
2998-
2999-static float stbir__support_two(float s, void * user_data)
3000-{
3001- STBIR__UNUSED(s);
3002- STBIR__UNUSED(user_data);
3003- return 2;
3004-}
3005-
3006-// This is the maximum number of input samples that can affect an output sample
3007-// with the given filter from the output pixel's perspective
3008-static int stbir__get_filter_pixel_width(stbir__support_callback * support, float scale, void * user_data)
3009-{
3010- STBIR_ASSERT(support != 0);
3011-
3012- if ( scale >= ( 1.0f-stbir__small_float ) ) // upscale
3013- return (int)STBIR_CEILF(support(1.0f/scale,user_data) * 2.0f);
3014- else
3015- return (int)STBIR_CEILF(support(scale,user_data) * 2.0f / scale);
3016-}
3017-
3018-// this is how many coefficents per run of the filter (which is different
3019-// from the filter_pixel_width depending on if we are scattering or gathering)
3020-static int stbir__get_coefficient_width(stbir__sampler * samp, int is_gather, void * user_data)
3021-{
3022- float scale = samp->scale_info.scale;
3023- stbir__support_callback * support = samp->filter_support;
3024-
3025- switch( is_gather )
3026- {
3027- case 1:
3028- return (int)STBIR_CEILF(support(1.0f / scale, user_data) * 2.0f);
3029- case 2:
3030- return (int)STBIR_CEILF(support(scale, user_data) * 2.0f / scale);
3031- case 0:
3032- return (int)STBIR_CEILF(support(scale, user_data) * 2.0f);
3033- default:
3034- STBIR_ASSERT( (is_gather >= 0 ) && (is_gather <= 2 ) );
3035- return 0;
3036- }
3037-}
3038-
3039-static int stbir__get_contributors(stbir__sampler * samp, int is_gather)
3040-{
3041- if (is_gather)
3042- return samp->scale_info.output_sub_size;
3043- else
3044- return (samp->scale_info.input_full_size + samp->filter_pixel_margin * 2);
3045-}
3046-
3047-static int stbir__edge_zero_full( int n, int max )
3048-{
3049- STBIR__UNUSED(n);
3050- STBIR__UNUSED(max);
3051- return 0; // NOTREACHED
3052-}
3053-
3054-static int stbir__edge_clamp_full( int n, int max )
3055-{
3056- if (n < 0)
3057- return 0;
3058-
3059- if (n >= max)
3060- return max - 1;
3061-
3062- return n; // NOTREACHED
3063-}
3064-
3065-static int stbir__edge_reflect_full( int n, int max )
3066-{
3067- if (n < 0)
3068- {
3069- if (n > -max)
3070- return -n;
3071- else
3072- return max - 1;
3073- }
3074-
3075- if (n >= max)
3076- {
3077- int max2 = max * 2;
3078- if (n >= max2)
3079- return 0;
3080- else
3081- return max2 - n - 1;
3082- }
3083-
3084- return n; // NOTREACHED
3085-}
3086-
3087-static int stbir__edge_wrap_full( int n, int max )
3088-{
3089- if (n >= 0)
3090- return (n % max);
3091- else
3092- {
3093- int m = (-n) % max;
3094-
3095- if (m != 0)
3096- m = max - m;
3097-
3098- return (m);
3099- }
3100-}
3101-
3102-typedef int stbir__edge_wrap_func( int n, int max );
3103-static stbir__edge_wrap_func * stbir__edge_wrap_slow[] =
3104-{
3105- stbir__edge_clamp_full, // STBIR_EDGE_CLAMP
3106- stbir__edge_reflect_full, // STBIR_EDGE_REFLECT
3107- stbir__edge_wrap_full, // STBIR_EDGE_WRAP
3108- stbir__edge_zero_full, // STBIR_EDGE_ZERO
3109-};
3110-
3111-stbir__inline static int stbir__edge_wrap(stbir_edge edge, int n, int max)
3112-{
3113- // avoid per-pixel switch
3114- if (n >= 0 && n < max)
3115- return n;
3116- return stbir__edge_wrap_slow[edge]( n, max );
3117-}
3118-
3119-#define STBIR__MERGE_RUNS_PIXEL_THRESHOLD 16
3120-
3121-// get information on the extents of a sampler
3122-static void stbir__get_extents( stbir__sampler * samp, stbir__extents * scanline_extents )
3123-{
3124- int j, stop;
3125- int left_margin, right_margin;
3126- int min_n = 0x7fffffff, max_n = -0x7fffffff;
3127- int min_left = 0x7fffffff, max_left = -0x7fffffff;
3128- int min_right = 0x7fffffff, max_right = -0x7fffffff;
3129- stbir_edge edge = samp->edge;
3130- stbir__contributors* contributors = samp->contributors;
3131- int output_sub_size = samp->scale_info.output_sub_size;
3132- int input_full_size = samp->scale_info.input_full_size;
3133- int filter_pixel_margin = samp->filter_pixel_margin;
3134-
3135- STBIR_ASSERT( samp->is_gather );
3136-
3137- stop = output_sub_size;
3138- for (j = 0; j < stop; j++ )
3139- {
3140- STBIR_ASSERT( contributors[j].n1 >= contributors[j].n0 );
3141- if ( contributors[j].n0 < min_n )
3142- {
3143- min_n = contributors[j].n0;
3144- stop = j + filter_pixel_margin; // if we find a new min, only scan another filter width
3145- if ( stop > output_sub_size ) stop = output_sub_size;
3146- }
3147- }
3148-
3149- stop = 0;
3150- for (j = output_sub_size - 1; j >= stop; j-- )
3151- {
3152- STBIR_ASSERT( contributors[j].n1 >= contributors[j].n0 );
3153- if ( contributors[j].n1 > max_n )
3154- {
3155- max_n = contributors[j].n1;
3156- stop = j - filter_pixel_margin; // if we find a new max, only scan another filter width
3157- if (stop<0) stop = 0;
3158- }
3159- }
3160-
3161- STBIR_ASSERT( scanline_extents->conservative.n0 <= min_n );
3162- STBIR_ASSERT( scanline_extents->conservative.n1 >= max_n );
3163-
3164- // now calculate how much into the margins we really read
3165- left_margin = 0;
3166- if ( min_n < 0 )
3167- {
3168- left_margin = -min_n;
3169- min_n = 0;
3170- }
3171-
3172- right_margin = 0;
3173- if ( max_n >= input_full_size )
3174- {
3175- right_margin = max_n - input_full_size + 1;
3176- max_n = input_full_size - 1;
3177- }
3178-
3179- // index 1 is margin pixel extents (how many pixels we hang over the edge)
3180- scanline_extents->edge_sizes[0] = left_margin;
3181- scanline_extents->edge_sizes[1] = right_margin;
3182-
3183- // index 2 is pixels read from the input
3184- scanline_extents->spans[0].n0 = min_n;
3185- scanline_extents->spans[0].n1 = max_n;
3186- scanline_extents->spans[0].pixel_offset_for_input = min_n;
3187-
3188- // default to no other input range
3189- scanline_extents->spans[1].n0 = 0;
3190- scanline_extents->spans[1].n1 = -1;
3191- scanline_extents->spans[1].pixel_offset_for_input = 0;
3192-
3193- // don't have to do edge calc for zero clamp
3194- if ( edge == STBIR_EDGE_ZERO )
3195- return;
3196-
3197- // convert margin pixels to the pixels within the input (min and max)
3198- for( j = -left_margin ; j < 0 ; j++ )
3199- {
3200- int p = stbir__edge_wrap( edge, j, input_full_size );
3201- if ( p < min_left )
3202- min_left = p;
3203- if ( p > max_left )
3204- max_left = p;
3205- }
3206-
3207- for( j = input_full_size ; j < (input_full_size + right_margin) ; j++ )
3208- {
3209- int p = stbir__edge_wrap( edge, j, input_full_size );
3210- if ( p < min_right )
3211- min_right = p;
3212- if ( p > max_right )
3213- max_right = p;
3214- }
3215-
3216- // merge the left margin pixel region if it connects within 4 pixels of main pixel region
3217- if ( min_left != 0x7fffffff )
3218- {
3219- if ( ( ( min_left <= min_n ) && ( ( max_left + STBIR__MERGE_RUNS_PIXEL_THRESHOLD ) >= min_n ) ) ||
3220- ( ( min_n <= min_left ) && ( ( max_n + STBIR__MERGE_RUNS_PIXEL_THRESHOLD ) >= max_left ) ) )
3221- {
3222- scanline_extents->spans[0].n0 = min_n = stbir__min( min_n, min_left );
3223- scanline_extents->spans[0].n1 = max_n = stbir__max( max_n, max_left );
3224- scanline_extents->spans[0].pixel_offset_for_input = min_n;
3225- left_margin = 0;
3226- }
3227- }
3228-
3229- // merge the right margin pixel region if it connects within 4 pixels of main pixel region
3230- if ( min_right != 0x7fffffff )
3231- {
3232- if ( ( ( min_right <= min_n ) && ( ( max_right + STBIR__MERGE_RUNS_PIXEL_THRESHOLD ) >= min_n ) ) ||
3233- ( ( min_n <= min_right ) && ( ( max_n + STBIR__MERGE_RUNS_PIXEL_THRESHOLD ) >= max_right ) ) )
3234- {
3235- scanline_extents->spans[0].n0 = min_n = stbir__min( min_n, min_right );
3236- scanline_extents->spans[0].n1 = max_n = stbir__max( max_n, max_right );
3237- scanline_extents->spans[0].pixel_offset_for_input = min_n;
3238- right_margin = 0;
3239- }
3240- }
3241-
3242- STBIR_ASSERT( scanline_extents->conservative.n0 <= min_n );
3243- STBIR_ASSERT( scanline_extents->conservative.n1 >= max_n );
3244-
3245- // you get two ranges when you have the WRAP edge mode and you are doing just the a piece of the resize
3246- // so you need to get a second run of pixels from the opposite side of the scanline (which you
3247- // wouldn't need except for WRAP)
3248-
3249-
3250- // if we can't merge the min_left range, add it as a second range
3251- if ( ( left_margin ) && ( min_left != 0x7fffffff ) )
3252- {
3253- stbir__span * newspan = scanline_extents->spans + 1;
3254- STBIR_ASSERT( right_margin == 0 );
3255- if ( min_left < scanline_extents->spans[0].n0 )
3256- {
3257- scanline_extents->spans[1].pixel_offset_for_input = scanline_extents->spans[0].n0;
3258- scanline_extents->spans[1].n0 = scanline_extents->spans[0].n0;
3259- scanline_extents->spans[1].n1 = scanline_extents->spans[0].n1;
3260- --newspan;
3261- }
3262- newspan->pixel_offset_for_input = min_left;
3263- newspan->n0 = -left_margin;
3264- newspan->n1 = ( max_left - min_left ) - left_margin;
3265- scanline_extents->edge_sizes[0] = 0; // don't need to copy the left margin, since we are directly decoding into the margin
3266- }
3267- // if we can't merge the min_right range, add it as a second range
3268- else
3269- if ( ( right_margin ) && ( min_right != 0x7fffffff ) )
3270- {
3271- stbir__span * newspan = scanline_extents->spans + 1;
3272- if ( min_right < scanline_extents->spans[0].n0 )
3273- {
3274- scanline_extents->spans[1].pixel_offset_for_input = scanline_extents->spans[0].n0;
3275- scanline_extents->spans[1].n0 = scanline_extents->spans[0].n0;
3276- scanline_extents->spans[1].n1 = scanline_extents->spans[0].n1;
3277- --newspan;
3278- }
3279- newspan->pixel_offset_for_input = min_right;
3280- newspan->n0 = scanline_extents->spans[1].n1 + 1;
3281- newspan->n1 = scanline_extents->spans[1].n1 + 1 + ( max_right - min_right );
3282- scanline_extents->edge_sizes[1] = 0; // don't need to copy the right margin, since we are directly decoding into the margin
3283- }
3284-
3285- // sort the spans into write output order
3286- if ( ( scanline_extents->spans[1].n1 > scanline_extents->spans[1].n0 ) && ( scanline_extents->spans[0].n0 > scanline_extents->spans[1].n0 ) )
3287- {
3288- stbir__span tspan = scanline_extents->spans[0];
3289- scanline_extents->spans[0] = scanline_extents->spans[1];
3290- scanline_extents->spans[1] = tspan;
3291- }
3292-}
3293-
3294-static void stbir__calculate_in_pixel_range( int * first_pixel, int * last_pixel, float out_pixel_center, float out_filter_radius, float inv_scale, float out_shift, int input_size, stbir_edge edge )
3295-{
3296- int first, last;
3297- float out_pixel_influence_lowerbound = out_pixel_center - out_filter_radius;
3298- float out_pixel_influence_upperbound = out_pixel_center + out_filter_radius;
3299-
3300- float in_pixel_influence_lowerbound = (out_pixel_influence_lowerbound + out_shift) * inv_scale;
3301- float in_pixel_influence_upperbound = (out_pixel_influence_upperbound + out_shift) * inv_scale;
3302-
3303- first = (int)(STBIR_FLOORF(in_pixel_influence_lowerbound + 0.5f));
3304- last = (int)(STBIR_FLOORF(in_pixel_influence_upperbound - 0.5f));
3305- if ( last < first ) last = first; // point sample mode can span a value *right* at 0.5, and cause these to cross
3306-
3307- if ( edge == STBIR_EDGE_WRAP )
3308- {
3309- if ( first < -input_size )
3310- first = -input_size;
3311- if ( last >= (input_size*2))
3312- last = (input_size*2) - 1;
3313- }
3314-
3315- *first_pixel = first;
3316- *last_pixel = last;
3317-}
3318-
3319-static void stbir__calculate_coefficients_for_gather_upsample( float out_filter_radius, stbir__kernel_callback * kernel, stbir__scale_info * scale_info, int num_contributors, stbir__contributors* contributors, float* coefficient_group, int coefficient_width, stbir_edge edge, void * user_data )
3320-{
3321- int n, end;
3322- float inv_scale = scale_info->inv_scale;
3323- float out_shift = scale_info->pixel_shift;
3324- int input_size = scale_info->input_full_size;
3325- int numerator = scale_info->scale_numerator;
3326- int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < num_contributors ) );
3327-
3328- // Looping through out pixels
3329- end = num_contributors; if ( polyphase ) end = numerator;
3330- for (n = 0; n < end; n++)
3331- {
3332- int i;
3333- int last_non_zero;
3334- float out_pixel_center = (float)n + 0.5f;
3335- float in_center_of_out = (out_pixel_center + out_shift) * inv_scale;
3336-
3337- int in_first_pixel, in_last_pixel;
3338-
3339- stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, out_pixel_center, out_filter_radius, inv_scale, out_shift, input_size, edge );
3340-
3341- // make sure we never generate a range larger than our precalculated coeff width
3342- // this only happens in point sample mode, but it's a good safe thing to do anyway
3343- if ( ( in_last_pixel - in_first_pixel + 1 ) > coefficient_width )
3344- in_last_pixel = in_first_pixel + coefficient_width - 1;
3345-
3346- last_non_zero = -1;
3347- for (i = 0; i <= in_last_pixel - in_first_pixel; i++)
3348- {
3349- float in_pixel_center = (float)(i + in_first_pixel) + 0.5f;
3350- float coeff = kernel(in_center_of_out - in_pixel_center, inv_scale, user_data);
3351-
3352- // kill denormals
3353- if ( ( ( coeff < stbir__small_float ) && ( coeff > -stbir__small_float ) ) )
3354- {
3355- if ( i == 0 ) // if we're at the front, just eat zero contributors
3356- {
3357- STBIR_ASSERT ( ( in_last_pixel - in_first_pixel ) != 0 ); // there should be at least one contrib
3358- ++in_first_pixel;
3359- i--;
3360- continue;
3361- }
3362- coeff = 0; // make sure is fully zero (should keep denormals away)
3363- }
3364- else
3365- last_non_zero = i;
3366-
3367- coefficient_group[i] = coeff;
3368- }
3369-
3370- in_last_pixel = last_non_zero+in_first_pixel; // kills trailing zeros
3371- contributors->n0 = in_first_pixel;
3372- contributors->n1 = in_last_pixel;
3373-
3374- STBIR_ASSERT(contributors->n1 >= contributors->n0);
3375-
3376- ++contributors;
3377- coefficient_group += coefficient_width;
3378- }
3379-}
3380-
3381-static void stbir__insert_coeff( stbir__contributors * contribs, float * coeffs, int new_pixel, float new_coeff, int max_width )
3382-{
3383- if ( contribs->n1 < contribs->n0 ) // this first clause should never happen, but handle in case
3384- {
3385- contribs->n0 = contribs->n1 = new_pixel;
3386- coeffs[0] = new_coeff;
3387- }
3388- else if ( new_pixel <= contribs->n1 ) // before the end
3389- {
3390- if ( new_pixel < contribs->n0 ) // before the front?
3391- {
3392- if ( ( contribs->n1 - new_pixel + 1 ) <= max_width )
3393- {
3394- int j, o = contribs->n0 - new_pixel;
3395- for ( j = contribs->n1 - contribs->n0 ; j >= 0 ; j-- )
3396- coeffs[ j + o ] = coeffs[ j ];
3397- for ( j = 1 ; j < o ; j++ )
3398- coeffs[ j ] = 0;
3399- coeffs[ 0 ] = new_coeff;
3400- contribs->n0 = new_pixel;
3401- }
3402- }
3403- else
3404- {
3405- // add new weight to existing coeff if already there
3406- coeffs[ new_pixel - contribs->n0 ] += new_coeff;
3407- }
3408- }
3409- else
3410- {
3411- if ( ( new_pixel - contribs->n0 + 1 ) <= max_width )
3412- {
3413- int j, e = new_pixel - contribs->n0;
3414- for( j = ( contribs->n1 - contribs->n0 ) + 1 ; j < e ; j++ ) // clear in-betweens coeffs if there are any
3415- coeffs[j] = 0;
3416-
3417- coeffs[ e ] = new_coeff;
3418- contribs->n1 = new_pixel;
3419- }
3420- }
3421-}
3422-
3423-static void stbir__calculate_out_pixel_range( int * first_pixel, int * last_pixel, float in_pixel_center, float in_pixels_radius, float scale, float out_shift, int out_size )
3424-{
3425- float in_pixel_influence_lowerbound = in_pixel_center - in_pixels_radius;
3426- float in_pixel_influence_upperbound = in_pixel_center + in_pixels_radius;
3427- float out_pixel_influence_lowerbound = in_pixel_influence_lowerbound * scale - out_shift;
3428- float out_pixel_influence_upperbound = in_pixel_influence_upperbound * scale - out_shift;
3429- int out_first_pixel = (int)(STBIR_FLOORF(out_pixel_influence_lowerbound + 0.5f));
3430- int out_last_pixel = (int)(STBIR_FLOORF(out_pixel_influence_upperbound - 0.5f));
3431-
3432- if ( out_first_pixel < 0 )
3433- out_first_pixel = 0;
3434- if ( out_last_pixel >= out_size )
3435- out_last_pixel = out_size - 1;
3436- *first_pixel = out_first_pixel;
3437- *last_pixel = out_last_pixel;
3438-}
3439-
3440-static void stbir__calculate_coefficients_for_gather_downsample( int start, int end, float in_pixels_radius, stbir__kernel_callback * kernel, stbir__scale_info * scale_info, int coefficient_width, int num_contributors, stbir__contributors * contributors, float * coefficient_group, void * user_data )
3441-{
3442- int in_pixel;
3443- int i;
3444- int first_out_inited = -1;
3445- float scale = scale_info->scale;
3446- float out_shift = scale_info->pixel_shift;
3447- int out_size = scale_info->output_sub_size;
3448- int numerator = scale_info->scale_numerator;
3449- int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < out_size ) );
3450-
3451- STBIR__UNUSED(num_contributors);
3452-
3453- // Loop through the input pixels
3454- for (in_pixel = start; in_pixel < end; in_pixel++)
3455- {
3456- float in_pixel_center = (float)in_pixel + 0.5f;
3457- float out_center_of_in = in_pixel_center * scale - out_shift;
3458- int out_first_pixel, out_last_pixel;
3459-
3460- stbir__calculate_out_pixel_range( &out_first_pixel, &out_last_pixel, in_pixel_center, in_pixels_radius, scale, out_shift, out_size );
3461-
3462- if ( out_first_pixel > out_last_pixel )
3463- continue;
3464-
3465- // clamp or exit if we are using polyphase filtering, and the limit is up
3466- if ( polyphase )
3467- {
3468- // when polyphase, you only have to do coeffs up to the numerator count
3469- if ( out_first_pixel == numerator )
3470- break;
3471-
3472- // don't do any extra work, clamp last pixel at numerator too
3473- if ( out_last_pixel >= numerator )
3474- out_last_pixel = numerator - 1;
3475- }
3476-
3477- for (i = 0; i <= out_last_pixel - out_first_pixel; i++)
3478- {
3479- float out_pixel_center = (float)(i + out_first_pixel) + 0.5f;
3480- float x = out_pixel_center - out_center_of_in;
3481- float coeff = kernel(x, scale, user_data) * scale;
3482-
3483- // kill the coeff if it's too small (avoid denormals)
3484- if ( ( ( coeff < stbir__small_float ) && ( coeff > -stbir__small_float ) ) )
3485- coeff = 0.0f;
3486-
3487- {
3488- int out = i + out_first_pixel;
3489- float * coeffs = coefficient_group + out * coefficient_width;
3490- stbir__contributors * contribs = contributors + out;
3491-
3492- // is this the first time this output pixel has been seen? Init it.
3493- if ( out > first_out_inited )
3494- {
3495- STBIR_ASSERT( out == ( first_out_inited + 1 ) ); // ensure we have only advanced one at time
3496- first_out_inited = out;
3497- contribs->n0 = in_pixel;
3498- contribs->n1 = in_pixel;
3499- coeffs[0] = coeff;
3500- }
3501- else
3502- {
3503- // insert on end (always in order)
3504- if ( coeffs[0] == 0.0f ) // if the first coefficent is zero, then zap it for this coeffs
3505- {
3506- STBIR_ASSERT( ( in_pixel - contribs->n0 ) == 1 ); // ensure that when we zap, we're at the 2nd pos
3507- contribs->n0 = in_pixel;
3508- }
3509- contribs->n1 = in_pixel;
3510- STBIR_ASSERT( ( in_pixel - contribs->n0 ) < coefficient_width );
3511- coeffs[in_pixel - contribs->n0] = coeff;
3512- }
3513- }
3514- }
3515- }
3516-}
3517-
3518-#ifdef STBIR_RENORMALIZE_IN_FLOAT
3519-#define STBIR_RENORM_TYPE float
3520-#else
3521-#define STBIR_RENORM_TYPE double
3522-#endif
3523-
3524-static void stbir__cleanup_gathered_coefficients( stbir_edge edge, stbir__filter_extent_info* filter_info, stbir__scale_info * scale_info, int num_contributors, stbir__contributors* contributors, float * coefficient_group, int coefficient_width )
3525-{
3526- int input_size = scale_info->input_full_size;
3527- int input_last_n1 = input_size - 1;
3528- int n, end;
3529- int lowest = 0x7fffffff;
3530- int highest = -0x7fffffff;
3531- int widest = -1;
3532- int numerator = scale_info->scale_numerator;
3533- int denominator = scale_info->scale_denominator;
3534- int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < num_contributors ) );
3535- float * coeffs;
3536- stbir__contributors * contribs;
3537-
3538- // weight all the coeffs for each sample
3539- coeffs = coefficient_group;
3540- contribs = contributors;
3541- end = num_contributors; if ( polyphase ) end = numerator;
3542- for (n = 0; n < end; n++)
3543- {
3544- int i;
3545- STBIR_RENORM_TYPE filter_scale, total_filter = 0;
3546- int e;
3547-
3548- // add all contribs
3549- e = contribs->n1 - contribs->n0;
3550- for( i = 0 ; i <= e ; i++ )
3551- {
3552- total_filter += (STBIR_RENORM_TYPE) coeffs[i];
3553- STBIR_ASSERT( ( coeffs[i] >= -2.0f ) && ( coeffs[i] <= 2.0f ) ); // check for wonky weights
3554- }
3555-
3556- // rescale
3557- if ( ( total_filter < stbir__small_float ) && ( total_filter > -stbir__small_float ) )
3558- {
3559- // all coeffs are extremely small, just zero it
3560- contribs->n1 = contribs->n0;
3561- coeffs[0] = 0.0f;
3562- }
3563- else
3564- {
3565- // if the total isn't 1.0, rescale everything
3566- if ( ( total_filter < (1.0f-stbir__small_float) ) || ( total_filter > (1.0f+stbir__small_float) ) )
3567- {
3568- filter_scale = ((STBIR_RENORM_TYPE)1.0) / total_filter;
3569-
3570- // scale them all
3571- for (i = 0; i <= e; i++)
3572- coeffs[i] = (float) ( coeffs[i] * filter_scale );
3573- }
3574- }
3575- ++contribs;
3576- coeffs += coefficient_width;
3577- }
3578-
3579- // if we have a rational for the scale, we can exploit the polyphaseness to not calculate
3580- // most of the coefficients, so we copy them here
3581- if ( polyphase )
3582- {
3583- stbir__contributors * prev_contribs = contributors;
3584- stbir__contributors * cur_contribs = contributors + numerator;
3585-
3586- for( n = numerator ; n < num_contributors ; n++ )
3587- {
3588- cur_contribs->n0 = prev_contribs->n0 + denominator;
3589- cur_contribs->n1 = prev_contribs->n1 + denominator;
3590- ++cur_contribs;
3591- ++prev_contribs;
3592- }
3593- stbir_overlapping_memcpy( coefficient_group + numerator * coefficient_width, coefficient_group, ( num_contributors - numerator ) * coefficient_width * sizeof( coeffs[ 0 ] ) );
3594- }
3595-
3596- coeffs = coefficient_group;
3597- contribs = contributors;
3598-
3599- for (n = 0; n < num_contributors; n++)
3600- {
3601- int i;
3602-
3603- // in zero edge mode, just remove out of bounds contribs completely (since their weights are accounted for now)
3604- if ( edge == STBIR_EDGE_ZERO )
3605- {
3606- // shrink the right side if necessary
3607- if ( contribs->n1 > input_last_n1 )
3608- contribs->n1 = input_last_n1;
3609-
3610- // shrink the left side
3611- if ( contribs->n0 < 0 )
3612- {
3613- int j, left, skips = 0;
3614-
3615- skips = -contribs->n0;
3616- contribs->n0 = 0;
3617-
3618- // now move down the weights
3619- left = contribs->n1 - contribs->n0 + 1;
3620- if ( left > 0 )
3621- {
3622- for( j = 0 ; j < left ; j++ )
3623- coeffs[ j ] = coeffs[ j + skips ];
3624- }
3625- }
3626- }
3627- else if ( ( edge == STBIR_EDGE_CLAMP ) || ( edge == STBIR_EDGE_REFLECT ) )
3628- {
3629- // for clamp and reflect, calculate the true inbounds position (based on edge type) and just add that to the existing weight
3630-
3631- // right hand side first
3632- if ( contribs->n1 > input_last_n1 )
3633- {
3634- int start = contribs->n0;
3635- int endi = contribs->n1;
3636- contribs->n1 = input_last_n1;
3637- for( i = input_size; i <= endi; i++ )
3638- stbir__insert_coeff( contribs, coeffs, stbir__edge_wrap_slow[edge]( i, input_size ), coeffs[i-start], coefficient_width );
3639- }
3640-
3641- // now check left hand edge
3642- if ( contribs->n0 < 0 )
3643- {
3644- int save_n0;
3645- float save_n0_coeff;
3646- float * c = coeffs - ( contribs->n0 + 1 );
3647-
3648- // reinsert the coeffs with it reflected or clamped (insert accumulates, if the coeffs exist)
3649- for( i = -1 ; i > contribs->n0 ; i-- )
3650- stbir__insert_coeff( contribs, coeffs, stbir__edge_wrap_slow[edge]( i, input_size ), *c--, coefficient_width );
3651- save_n0 = contribs->n0;
3652- save_n0_coeff = c[0]; // save it, since we didn't do the final one (i==n0), because there might be too many coeffs to hold (before we resize)!
3653-
3654- // now slide all the coeffs down (since we have accumulated them in the positive contribs) and reset the first contrib
3655- contribs->n0 = 0;
3656- for(i = 0 ; i <= contribs->n1 ; i++ )
3657- coeffs[i] = coeffs[i-save_n0];
3658-
3659- // now that we have shrunk down the contribs, we insert the first one safely
3660- stbir__insert_coeff( contribs, coeffs, stbir__edge_wrap_slow[edge]( save_n0, input_size ), save_n0_coeff, coefficient_width );
3661- }
3662- }
3663-
3664- if ( contribs->n0 <= contribs->n1 )
3665- {
3666- int diff = contribs->n1 - contribs->n0 + 1;
3667- while ( diff && ( coeffs[ diff-1 ] == 0.0f ) )
3668- --diff;
3669-
3670- contribs->n1 = contribs->n0 + diff - 1;
3671-
3672- if ( contribs->n0 <= contribs->n1 )
3673- {
3674- if ( contribs->n0 < lowest )
3675- lowest = contribs->n0;
3676- if ( contribs->n1 > highest )
3677- highest = contribs->n1;
3678- if ( diff > widest )
3679- widest = diff;
3680- }
3681-
3682- // re-zero out unused coefficients (if any)
3683- for( i = diff ; i < coefficient_width ; i++ )
3684- coeffs[i] = 0.0f;
3685- }
3686-
3687- ++contribs;
3688- coeffs += coefficient_width;
3689- }
3690- filter_info->lowest = lowest;
3691- filter_info->highest = highest;
3692- filter_info->widest = widest;
3693-}
3694-
3695-#undef STBIR_RENORM_TYPE
3696-
3697-static int stbir__pack_coefficients( int num_contributors, stbir__contributors* contributors, float * coefficents, int coefficient_width, int widest, int row0, int row1 )
3698-{
3699- #define STBIR_MOVE_1( dest, src ) { STBIR_NO_UNROLL(dest); ((stbir_uint32*)(dest))[0] = ((stbir_uint32*)(src))[0]; }
3700- #define STBIR_MOVE_2( dest, src ) { STBIR_NO_UNROLL(dest); ((stbir_uint64*)(dest))[0] = ((stbir_uint64*)(src))[0]; }
3701- #ifdef STBIR_SIMD
3702- #define STBIR_MOVE_4( dest, src ) { stbir__simdf t; STBIR_NO_UNROLL(dest); stbir__simdf_load( t, src ); stbir__simdf_store( dest, t ); }
3703- #else
3704- #define STBIR_MOVE_4( dest, src ) { STBIR_NO_UNROLL(dest); ((stbir_uint64*)(dest))[0] = ((stbir_uint64*)(src))[0]; ((stbir_uint64*)(dest))[1] = ((stbir_uint64*)(src))[1]; }
3705- #endif
3706-
3707- int row_end = row1 + 1;
3708- STBIR__UNUSED( row0 ); // only used in an assert
3709-
3710- if ( coefficient_width != widest )
3711- {
3712- float * pc = coefficents;
3713- float * coeffs = coefficents;
3714- float * pc_end = coefficents + num_contributors * widest;
3715- switch( widest )
3716- {
3717- case 1:
3718- STBIR_NO_UNROLL_LOOP_START
3719- do {
3720- STBIR_MOVE_1( pc, coeffs );
3721- ++pc;
3722- coeffs += coefficient_width;
3723- } while ( pc < pc_end );
3724- break;
3725- case 2:
3726- STBIR_NO_UNROLL_LOOP_START
3727- do {
3728- STBIR_MOVE_2( pc, coeffs );
3729- pc += 2;
3730- coeffs += coefficient_width;
3731- } while ( pc < pc_end );
3732- break;
3733- case 3:
3734- STBIR_NO_UNROLL_LOOP_START
3735- do {
3736- STBIR_MOVE_2( pc, coeffs );
3737- STBIR_MOVE_1( pc+2, coeffs+2 );
3738- pc += 3;
3739- coeffs += coefficient_width;
3740- } while ( pc < pc_end );
3741- break;
3742- case 4:
3743- STBIR_NO_UNROLL_LOOP_START
3744- do {
3745- STBIR_MOVE_4( pc, coeffs );
3746- pc += 4;
3747- coeffs += coefficient_width;
3748- } while ( pc < pc_end );
3749- break;
3750- case 5:
3751- STBIR_NO_UNROLL_LOOP_START
3752- do {
3753- STBIR_MOVE_4( pc, coeffs );
3754- STBIR_MOVE_1( pc+4, coeffs+4 );
3755- pc += 5;
3756- coeffs += coefficient_width;
3757- } while ( pc < pc_end );
3758- break;
3759- case 6:
3760- STBIR_NO_UNROLL_LOOP_START
3761- do {
3762- STBIR_MOVE_4( pc, coeffs );
3763- STBIR_MOVE_2( pc+4, coeffs+4 );
3764- pc += 6;
3765- coeffs += coefficient_width;
3766- } while ( pc < pc_end );
3767- break;
3768- case 7:
3769- STBIR_NO_UNROLL_LOOP_START
3770- do {
3771- STBIR_MOVE_4( pc, coeffs );
3772- STBIR_MOVE_2( pc+4, coeffs+4 );
3773- STBIR_MOVE_1( pc+6, coeffs+6 );
3774- pc += 7;
3775- coeffs += coefficient_width;
3776- } while ( pc < pc_end );
3777- break;
3778- case 8:
3779- STBIR_NO_UNROLL_LOOP_START
3780- do {
3781- STBIR_MOVE_4( pc, coeffs );
3782- STBIR_MOVE_4( pc+4, coeffs+4 );
3783- pc += 8;
3784- coeffs += coefficient_width;
3785- } while ( pc < pc_end );
3786- break;
3787- case 9:
3788- STBIR_NO_UNROLL_LOOP_START
3789- do {
3790- STBIR_MOVE_4( pc, coeffs );
3791- STBIR_MOVE_4( pc+4, coeffs+4 );
3792- STBIR_MOVE_1( pc+8, coeffs+8 );
3793- pc += 9;
3794- coeffs += coefficient_width;
3795- } while ( pc < pc_end );
3796- break;
3797- case 10:
3798- STBIR_NO_UNROLL_LOOP_START
3799- do {
3800- STBIR_MOVE_4( pc, coeffs );
3801- STBIR_MOVE_4( pc+4, coeffs+4 );
3802- STBIR_MOVE_2( pc+8, coeffs+8 );
3803- pc += 10;
3804- coeffs += coefficient_width;
3805- } while ( pc < pc_end );
3806- break;
3807- case 11:
3808- STBIR_NO_UNROLL_LOOP_START
3809- do {
3810- STBIR_MOVE_4( pc, coeffs );
3811- STBIR_MOVE_4( pc+4, coeffs+4 );
3812- STBIR_MOVE_2( pc+8, coeffs+8 );
3813- STBIR_MOVE_1( pc+10, coeffs+10 );
3814- pc += 11;
3815- coeffs += coefficient_width;
3816- } while ( pc < pc_end );
3817- break;
3818- case 12:
3819- STBIR_NO_UNROLL_LOOP_START
3820- do {
3821- STBIR_MOVE_4( pc, coeffs );
3822- STBIR_MOVE_4( pc+4, coeffs+4 );
3823- STBIR_MOVE_4( pc+8, coeffs+8 );
3824- pc += 12;
3825- coeffs += coefficient_width;
3826- } while ( pc < pc_end );
3827- break;
3828- default:
3829- STBIR_NO_UNROLL_LOOP_START
3830- do {
3831- float * copy_end = pc + widest - 4;
3832- float * c = coeffs;
3833- do {
3834- STBIR_NO_UNROLL( pc );
3835- STBIR_MOVE_4( pc, c );
3836- pc += 4;
3837- c += 4;
3838- } while ( pc <= copy_end );
3839- copy_end += 4;
3840- STBIR_NO_UNROLL_LOOP_START
3841- while ( pc < copy_end )
3842- {
3843- STBIR_MOVE_1( pc, c );
3844- ++pc; ++c;
3845- }
3846- coeffs += coefficient_width;
3847- } while ( pc < pc_end );
3848- break;
3849- }
3850- }
3851-
3852- // some horizontal routines read one float off the end (which is then masked off), so put in a sentinel so we don't read an snan or denormal
3853- coefficents[ widest * num_contributors ] = 8888.0f;
3854-
3855- // the minimum we might read for unrolled filters widths is 12. So, we need to
3856- // make sure we never read outside the decode buffer, by possibly moving
3857- // the sample area back into the scanline, and putting zeros weights first.
3858- // we start on the right edge and check until we're well past the possible
3859- // clip area (2*widest).
3860- {
3861- stbir__contributors * contribs = contributors + num_contributors - 1;
3862- float * coeffs = coefficents + widest * ( num_contributors - 1 );
3863-
3864- // go until no chance of clipping (this is usually less than 8 lops)
3865- while ( ( contribs >= contributors ) && ( ( contribs->n0 + widest*2 ) >= row_end ) )
3866- {
3867- // might we clip??
3868- if ( ( contribs->n0 + widest ) > row_end )
3869- {
3870- int stop_range = widest;
3871-
3872- // if range is larger than 12, it will be handled by generic loops that can terminate on the exact length
3873- // of this contrib n1, instead of a fixed widest amount - so calculate this
3874- if ( widest > 12 )
3875- {
3876- int mod;
3877-
3878- // how far will be read in the n_coeff loop (which depends on the widest count mod4);
3879- mod = widest & 3;
3880- stop_range = ( ( ( contribs->n1 - contribs->n0 + 1 ) - mod + 3 ) & ~3 ) + mod;
3881-
3882- // the n_coeff loops do a minimum amount of coeffs, so factor that in!
3883- if ( stop_range < ( 8 + mod ) ) stop_range = 8 + mod;
3884- }
3885-
3886- // now see if we still clip with the refined range
3887- if ( ( contribs->n0 + stop_range ) > row_end )
3888- {
3889- int new_n0 = row_end - stop_range;
3890- int num = contribs->n1 - contribs->n0 + 1;
3891- int backup = contribs->n0 - new_n0;
3892- float * from_co = coeffs + num - 1;
3893- float * to_co = from_co + backup;
3894-
3895- STBIR_ASSERT( ( new_n0 >= row0 ) && ( new_n0 < contribs->n0 ) );
3896-
3897- // move the coeffs over
3898- while( num )
3899- {
3900- *to_co-- = *from_co--;
3901- --num;
3902- }
3903- // zero new positions
3904- while ( to_co >= coeffs )
3905- *to_co-- = 0;
3906- // set new start point
3907- contribs->n0 = new_n0;
3908- if ( widest > 12 )
3909- {
3910- int mod;
3911-
3912- // how far will be read in the n_coeff loop (which depends on the widest count mod4);
3913- mod = widest & 3;
3914- stop_range = ( ( ( contribs->n1 - contribs->n0 + 1 ) - mod + 3 ) & ~3 ) + mod;
3915-
3916- // the n_coeff loops do a minimum amount of coeffs, so factor that in!
3917- if ( stop_range < ( 8 + mod ) ) stop_range = 8 + mod;
3918- }
3919- }
3920- }
3921- --contribs;
3922- coeffs -= widest;
3923- }
3924- }
3925-
3926- return widest;
3927- #undef STBIR_MOVE_1
3928- #undef STBIR_MOVE_2
3929- #undef STBIR_MOVE_4
3930-}
3931-
3932-static void stbir__calculate_filters( stbir__sampler * samp, stbir__sampler * other_axis_for_pivot, void * user_data STBIR_ONLY_PROFILE_BUILD_GET_INFO )
3933-{
3934- int n;
3935- float scale = samp->scale_info.scale;
3936- stbir__kernel_callback * kernel = samp->filter_kernel;
3937- stbir__support_callback * support = samp->filter_support;
3938- float inv_scale = samp->scale_info.inv_scale;
3939- int input_full_size = samp->scale_info.input_full_size;
3940- int gather_num_contributors = samp->num_contributors;
3941- stbir__contributors* gather_contributors = samp->contributors;
3942- float * gather_coeffs = samp->coefficients;
3943- int gather_coefficient_width = samp->coefficient_width;
3944-
3945- switch ( samp->is_gather )
3946- {
3947- case 1: // gather upsample
3948- {
3949- float out_pixels_radius = support(inv_scale,user_data) * scale;
3950-
3951- stbir__calculate_coefficients_for_gather_upsample( out_pixels_radius, kernel, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width, samp->edge, user_data );
3952-
3953- STBIR_PROFILE_BUILD_START( cleanup );
3954- stbir__cleanup_gathered_coefficients( samp->edge, &samp->extent_info, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width );
3955- STBIR_PROFILE_BUILD_END( cleanup );
3956- }
3957- break;
3958-
3959- case 0: // scatter downsample (only on vertical)
3960- case 2: // gather downsample
3961- {
3962- float in_pixels_radius = support(scale,user_data) * inv_scale;
3963- int filter_pixel_margin = samp->filter_pixel_margin;
3964- int input_end = input_full_size + filter_pixel_margin;
3965-
3966- // if this is a scatter, we do a downsample gather to get the coeffs, and then pivot after
3967- if ( !samp->is_gather )
3968- {
3969- // check if we are using the same gather downsample on the horizontal as this vertical,
3970- // if so, then we don't have to generate them, we can just pivot from the horizontal.
3971- if ( other_axis_for_pivot )
3972- {
3973- gather_contributors = other_axis_for_pivot->contributors;
3974- gather_coeffs = other_axis_for_pivot->coefficients;
3975- gather_coefficient_width = other_axis_for_pivot->coefficient_width;
3976- gather_num_contributors = other_axis_for_pivot->num_contributors;
3977- samp->extent_info.lowest = other_axis_for_pivot->extent_info.lowest;
3978- samp->extent_info.highest = other_axis_for_pivot->extent_info.highest;
3979- samp->extent_info.widest = other_axis_for_pivot->extent_info.widest;
3980- goto jump_right_to_pivot;
3981- }
3982-
3983- gather_contributors = samp->gather_prescatter_contributors;
3984- gather_coeffs = samp->gather_prescatter_coefficients;
3985- gather_coefficient_width = samp->gather_prescatter_coefficient_width;
3986- gather_num_contributors = samp->gather_prescatter_num_contributors;
3987- }
3988-
3989- stbir__calculate_coefficients_for_gather_downsample( -filter_pixel_margin, input_end, in_pixels_radius, kernel, &samp->scale_info, gather_coefficient_width, gather_num_contributors, gather_contributors, gather_coeffs, user_data );
3990-
3991- STBIR_PROFILE_BUILD_START( cleanup );
3992- stbir__cleanup_gathered_coefficients( samp->edge, &samp->extent_info, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width );
3993- STBIR_PROFILE_BUILD_END( cleanup );
3994-
3995- if ( !samp->is_gather )
3996- {
3997- // if this is a scatter (vertical only), then we need to pivot the coeffs
3998- stbir__contributors * scatter_contributors;
3999- int highest_set;
4000-
4001- jump_right_to_pivot:
4002-
4003- STBIR_PROFILE_BUILD_START( pivot );
4004-
4005- highest_set = (-filter_pixel_margin) - 1;
4006- for (n = 0; n < gather_num_contributors; n++)
4007- {
4008- int k;
4009- int gn0 = gather_contributors->n0, gn1 = gather_contributors->n1;
4010- int scatter_coefficient_width = samp->coefficient_width;
4011- float * scatter_coeffs = samp->coefficients + ( gn0 + filter_pixel_margin ) * scatter_coefficient_width;
4012- float * g_coeffs = gather_coeffs;
4013- scatter_contributors = samp->contributors + ( gn0 + filter_pixel_margin );
4014-
4015- for (k = gn0 ; k <= gn1 ; k++ )
4016- {
4017- float gc = *g_coeffs++;
4018-
4019- // skip zero and denormals - must skip zeros to avoid adding coeffs beyond scatter_coefficient_width
4020- // (which happens when pivoting from horizontal, which might have dummy zeros)
4021- if ( ( ( gc >= stbir__small_float ) || ( gc <= -stbir__small_float ) ) )
4022- {
4023- if ( ( k > highest_set ) || ( scatter_contributors->n0 > scatter_contributors->n1 ) )
4024- {
4025- {
4026- // if we are skipping over several contributors, we need to clear the skipped ones
4027- stbir__contributors * clear_contributors = samp->contributors + ( highest_set + filter_pixel_margin + 1);
4028- while ( clear_contributors < scatter_contributors )
4029- {
4030- clear_contributors->n0 = 0;
4031- clear_contributors->n1 = -1;
4032- ++clear_contributors;
4033- }
4034- }
4035- scatter_contributors->n0 = n;
4036- scatter_contributors->n1 = n;
4037- scatter_coeffs[0] = gc;
4038- highest_set = k;
4039- }
4040- else
4041- {
4042- stbir__insert_coeff( scatter_contributors, scatter_coeffs, n, gc, scatter_coefficient_width );
4043- }
4044- STBIR_ASSERT( ( scatter_contributors->n1 - scatter_contributors->n0 + 1 ) <= scatter_coefficient_width );
4045- }
4046- ++scatter_contributors;
4047- scatter_coeffs += scatter_coefficient_width;
4048- }
4049-
4050- ++gather_contributors;
4051- gather_coeffs += gather_coefficient_width;
4052- }
4053-
4054- // now clear any unset contribs
4055- {
4056- stbir__contributors * clear_contributors = samp->contributors + ( highest_set + filter_pixel_margin + 1);
4057- stbir__contributors * end_contributors = samp->contributors + samp->num_contributors;
4058- while ( clear_contributors < end_contributors )
4059- {
4060- clear_contributors->n0 = 0;
4061- clear_contributors->n1 = -1;
4062- ++clear_contributors;
4063- }
4064- }
4065-
4066- STBIR_PROFILE_BUILD_END( pivot );
4067- }
4068- }
4069- break;
4070- }
4071-}
4072-
4073-
4074-//========================================================================================================
4075-// scanline decoders and encoders
4076-
4077-#define stbir__coder_min_num 1
4078-#define STB_IMAGE_RESIZE_DO_CODERS
4079-#include STBIR__HEADER_FILENAME
4080-
4081-#define stbir__decode_suffix BGRA
4082-#define stbir__decode_swizzle
4083-#define stbir__decode_order0 2
4084-#define stbir__decode_order1 1
4085-#define stbir__decode_order2 0
4086-#define stbir__decode_order3 3
4087-#define stbir__encode_order0 2
4088-#define stbir__encode_order1 1
4089-#define stbir__encode_order2 0
4090-#define stbir__encode_order3 3
4091-#define stbir__coder_min_num 4
4092-#define STB_IMAGE_RESIZE_DO_CODERS
4093-#include STBIR__HEADER_FILENAME
4094-
4095-#define stbir__decode_suffix ARGB
4096-#define stbir__decode_swizzle
4097-#define stbir__decode_order0 1
4098-#define stbir__decode_order1 2
4099-#define stbir__decode_order2 3
4100-#define stbir__decode_order3 0
4101-#define stbir__encode_order0 3
4102-#define stbir__encode_order1 0
4103-#define stbir__encode_order2 1
4104-#define stbir__encode_order3 2
4105-#define stbir__coder_min_num 4
4106-#define STB_IMAGE_RESIZE_DO_CODERS
4107-#include STBIR__HEADER_FILENAME
4108-
4109-#define stbir__decode_suffix ABGR
4110-#define stbir__decode_swizzle
4111-#define stbir__decode_order0 3
4112-#define stbir__decode_order1 2
4113-#define stbir__decode_order2 1
4114-#define stbir__decode_order3 0
4115-#define stbir__encode_order0 3
4116-#define stbir__encode_order1 2
4117-#define stbir__encode_order2 1
4118-#define stbir__encode_order3 0
4119-#define stbir__coder_min_num 4
4120-#define STB_IMAGE_RESIZE_DO_CODERS
4121-#include STBIR__HEADER_FILENAME
4122-
4123-#define stbir__decode_suffix AR
4124-#define stbir__decode_swizzle
4125-#define stbir__decode_order0 1
4126-#define stbir__decode_order1 0
4127-#define stbir__decode_order2 3
4128-#define stbir__decode_order3 2
4129-#define stbir__encode_order0 1
4130-#define stbir__encode_order1 0
4131-#define stbir__encode_order2 3
4132-#define stbir__encode_order3 2
4133-#define stbir__coder_min_num 2
4134-#define STB_IMAGE_RESIZE_DO_CODERS
4135-#include STBIR__HEADER_FILENAME
4136-
4137-
4138-// fancy alpha means we expand to keep both premultipied and non-premultiplied color channels
4139-static void stbir__fancy_alpha_weight_4ch( float * out_buffer, int width_times_channels )
4140-{
4141- float STBIR_STREAMOUT_PTR(*) out = out_buffer;
4142- float const * end_decode = out_buffer + ( width_times_channels / 4 ) * 7; // decode buffer aligned to end of out_buffer
4143- float STBIR_STREAMOUT_PTR(*) decode = (float*)end_decode - width_times_channels;
4144-
4145- // fancy alpha is stored internally as R G B A Rpm Gpm Bpm
4146-
4147- #ifdef STBIR_SIMD
4148-
4149- #ifdef STBIR_SIMD8
4150- decode += 16;
4151- STBIR_NO_UNROLL_LOOP_START
4152- while ( decode <= end_decode )
4153- {
4154- stbir__simdf8 d0,d1,a0,a1,p0,p1;
4155- STBIR_NO_UNROLL(decode);
4156- stbir__simdf8_load( d0, decode-16 );
4157- stbir__simdf8_load( d1, decode-16+8 );
4158- stbir__simdf8_0123to33333333( a0, d0 );
4159- stbir__simdf8_0123to33333333( a1, d1 );
4160- stbir__simdf8_mult( p0, a0, d0 );
4161- stbir__simdf8_mult( p1, a1, d1 );
4162- stbir__simdf8_bot4s( a0, d0, p0 );
4163- stbir__simdf8_bot4s( a1, d1, p1 );
4164- stbir__simdf8_top4s( d0, d0, p0 );
4165- stbir__simdf8_top4s( d1, d1, p1 );
4166- stbir__simdf8_store ( out, a0 );
4167- stbir__simdf8_store ( out+7, d0 );
4168- stbir__simdf8_store ( out+14, a1 );
4169- stbir__simdf8_store ( out+21, d1 );
4170- decode += 16;
4171- out += 28;
4172- }
4173- decode -= 16;
4174- #else
4175- decode += 8;
4176- STBIR_NO_UNROLL_LOOP_START
4177- while ( decode <= end_decode )
4178- {
4179- stbir__simdf d0,a0,d1,a1,p0,p1;
4180- STBIR_NO_UNROLL(decode);
4181- stbir__simdf_load( d0, decode-8 );
4182- stbir__simdf_load( d1, decode-8+4 );
4183- stbir__simdf_0123to3333( a0, d0 );
4184- stbir__simdf_0123to3333( a1, d1 );
4185- stbir__simdf_mult( p0, a0, d0 );
4186- stbir__simdf_mult( p1, a1, d1 );
4187- stbir__simdf_store ( out, d0 );
4188- stbir__simdf_store ( out+4, p0 );
4189- stbir__simdf_store ( out+7, d1 );
4190- stbir__simdf_store ( out+7+4, p1 );
4191- decode += 8;
4192- out += 14;
4193- }
4194- decode -= 8;
4195- #endif
4196-
4197- // might be one last odd pixel
4198- #ifdef STBIR_SIMD8
4199- STBIR_NO_UNROLL_LOOP_START
4200- while ( decode < end_decode )
4201- #else
4202- if ( decode < end_decode )
4203- #endif
4204- {
4205- stbir__simdf d,a,p;
4206- STBIR_NO_UNROLL(decode);
4207- stbir__simdf_load( d, decode );
4208- stbir__simdf_0123to3333( a, d );
4209- stbir__simdf_mult( p, a, d );
4210- stbir__simdf_store ( out, d );
4211- stbir__simdf_store ( out+4, p );
4212- decode += 4;
4213- out += 7;
4214- }
4215-
4216- #else
4217-
4218- while( decode < end_decode )
4219- {
4220- float r = decode[0], g = decode[1], b = decode[2], alpha = decode[3];
4221- out[0] = r;
4222- out[1] = g;
4223- out[2] = b;
4224- out[3] = alpha;
4225- out[4] = r * alpha;
4226- out[5] = g * alpha;
4227- out[6] = b * alpha;
4228- out += 7;
4229- decode += 4;
4230- }
4231-
4232- #endif
4233-}
4234-
4235-static void stbir__fancy_alpha_weight_2ch( float * out_buffer, int width_times_channels )
4236-{
4237- float STBIR_STREAMOUT_PTR(*) out = out_buffer;
4238- float const * end_decode = out_buffer + ( width_times_channels / 2 ) * 3;
4239- float STBIR_STREAMOUT_PTR(*) decode = (float*)end_decode - width_times_channels;
4240-
4241- // for fancy alpha, turns into: [X A Xpm][X A Xpm],etc
4242-
4243- #ifdef STBIR_SIMD
4244-
4245- decode += 8;
4246- if ( decode <= end_decode )
4247- {
4248- STBIR_NO_UNROLL_LOOP_START
4249- do {
4250- #ifdef STBIR_SIMD8
4251- stbir__simdf8 d0,a0,p0;
4252- STBIR_NO_UNROLL(decode);
4253- stbir__simdf8_load( d0, decode-8 );
4254- stbir__simdf8_0123to11331133( p0, d0 );
4255- stbir__simdf8_0123to00220022( a0, d0 );
4256- stbir__simdf8_mult( p0, p0, a0 );
4257-
4258- stbir__simdf_store2( out, stbir__if_simdf8_cast_to_simdf4( d0 ) );
4259- stbir__simdf_store( out+2, stbir__if_simdf8_cast_to_simdf4( p0 ) );
4260- stbir__simdf_store2h( out+3, stbir__if_simdf8_cast_to_simdf4( d0 ) );
4261-
4262- stbir__simdf_store2( out+6, stbir__simdf8_gettop4( d0 ) );
4263- stbir__simdf_store( out+8, stbir__simdf8_gettop4( p0 ) );
4264- stbir__simdf_store2h( out+9, stbir__simdf8_gettop4( d0 ) );
4265- #else
4266- stbir__simdf d0,a0,d1,a1,p0,p1;
4267- STBIR_NO_UNROLL(decode);
4268- stbir__simdf_load( d0, decode-8 );
4269- stbir__simdf_load( d1, decode-8+4 );
4270- stbir__simdf_0123to1133( p0, d0 );
4271- stbir__simdf_0123to1133( p1, d1 );
4272- stbir__simdf_0123to0022( a0, d0 );
4273- stbir__simdf_0123to0022( a1, d1 );
4274- stbir__simdf_mult( p0, p0, a0 );
4275- stbir__simdf_mult( p1, p1, a1 );
4276-
4277- stbir__simdf_store2( out, d0 );
4278- stbir__simdf_store( out+2, p0 );
4279- stbir__simdf_store2h( out+3, d0 );
4280-
4281- stbir__simdf_store2( out+6, d1 );
4282- stbir__simdf_store( out+8, p1 );
4283- stbir__simdf_store2h( out+9, d1 );
4284- #endif
4285- decode += 8;
4286- out += 12;
4287- } while ( decode <= end_decode );
4288- }
4289- decode -= 8;
4290- #endif
4291-
4292- STBIR_SIMD_NO_UNROLL_LOOP_START
4293- while( decode < end_decode )
4294- {
4295- float x = decode[0], y = decode[1];
4296- STBIR_SIMD_NO_UNROLL(decode);
4297- out[0] = x;
4298- out[1] = y;
4299- out[2] = x * y;
4300- out += 3;
4301- decode += 2;
4302- }
4303-}
4304-
4305-static void stbir__fancy_alpha_unweight_4ch( float * encode_buffer, int width_times_channels )
4306-{
4307- float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
4308- float STBIR_SIMD_STREAMOUT_PTR(*) input = encode_buffer;
4309- float const * end_output = encode_buffer + width_times_channels;
4310-
4311- // fancy RGBA is stored internally as R G B A Rpm Gpm Bpm
4312-
4313- STBIR_SIMD_NO_UNROLL_LOOP_START
4314- do {
4315- float alpha = input[3];
4316-#ifdef STBIR_SIMD
4317- stbir__simdf i,ia;
4318- STBIR_SIMD_NO_UNROLL(encode);
4319- if ( alpha < stbir__small_float )
4320- {
4321- stbir__simdf_load( i, input );
4322- stbir__simdf_store( encode, i );
4323- }
4324- else
4325- {
4326- stbir__simdf_load1frep4( ia, 1.0f / alpha );
4327- stbir__simdf_load( i, input+4 );
4328- stbir__simdf_mult( i, i, ia );
4329- stbir__simdf_store( encode, i );
4330- encode[3] = alpha;
4331- }
4332-#else
4333- if ( alpha < stbir__small_float )
4334- {
4335- encode[0] = input[0];
4336- encode[1] = input[1];
4337- encode[2] = input[2];
4338- }
4339- else
4340- {
4341- float ialpha = 1.0f / alpha;
4342- encode[0] = input[4] * ialpha;
4343- encode[1] = input[5] * ialpha;
4344- encode[2] = input[6] * ialpha;
4345- }
4346- encode[3] = alpha;
4347-#endif
4348-
4349- input += 7;
4350- encode += 4;
4351- } while ( encode < end_output );
4352-}
4353-
4354-// format: [X A Xpm][X A Xpm] etc
4355-static void stbir__fancy_alpha_unweight_2ch( float * encode_buffer, int width_times_channels )
4356-{
4357- float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
4358- float STBIR_SIMD_STREAMOUT_PTR(*) input = encode_buffer;
4359- float const * end_output = encode_buffer + width_times_channels;
4360-
4361- do {
4362- float alpha = input[1];
4363- encode[0] = input[0];
4364- if ( alpha >= stbir__small_float )
4365- encode[0] = input[2] / alpha;
4366- encode[1] = alpha;
4367-
4368- input += 3;
4369- encode += 2;
4370- } while ( encode < end_output );
4371-}
4372-
4373-static void stbir__simple_alpha_weight_4ch( float * decode_buffer, int width_times_channels )
4374-{
4375- float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
4376- float const * end_decode = decode_buffer + width_times_channels;
4377-
4378- #ifdef STBIR_SIMD
4379- {
4380- decode += 2 * stbir__simdfX_float_count;
4381- STBIR_NO_UNROLL_LOOP_START
4382- while ( decode <= end_decode )
4383- {
4384- stbir__simdfX d0,a0,d1,a1;
4385- STBIR_NO_UNROLL(decode);
4386- stbir__simdfX_load( d0, decode-2*stbir__simdfX_float_count );
4387- stbir__simdfX_load( d1, decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count );
4388- stbir__simdfX_aaa1( a0, d0, STBIR_onesX );
4389- stbir__simdfX_aaa1( a1, d1, STBIR_onesX );
4390- stbir__simdfX_mult( d0, d0, a0 );
4391- stbir__simdfX_mult( d1, d1, a1 );
4392- stbir__simdfX_store ( decode-2*stbir__simdfX_float_count, d0 );
4393- stbir__simdfX_store ( decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count, d1 );
4394- decode += 2 * stbir__simdfX_float_count;
4395- }
4396- decode -= 2 * stbir__simdfX_float_count;
4397-
4398- // few last pixels remnants
4399- #ifdef STBIR_SIMD8
4400- STBIR_NO_UNROLL_LOOP_START
4401- while ( decode < end_decode )
4402- #else
4403- if ( decode < end_decode )
4404- #endif
4405- {
4406- stbir__simdf d,a;
4407- stbir__simdf_load( d, decode );
4408- stbir__simdf_aaa1( a, d, STBIR__CONSTF(STBIR_ones) );
4409- stbir__simdf_mult( d, d, a );
4410- stbir__simdf_store ( decode, d );
4411- decode += 4;
4412- }
4413- }
4414-
4415- #else
4416-
4417- while( decode < end_decode )
4418- {
4419- float alpha = decode[3];
4420- decode[0] *= alpha;
4421- decode[1] *= alpha;
4422- decode[2] *= alpha;
4423- decode += 4;
4424- }
4425-
4426- #endif
4427-}
4428-
4429-static void stbir__simple_alpha_weight_2ch( float * decode_buffer, int width_times_channels )
4430-{
4431- float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
4432- float const * end_decode = decode_buffer + width_times_channels;
4433-
4434- #ifdef STBIR_SIMD
4435- decode += 2 * stbir__simdfX_float_count;
4436- STBIR_NO_UNROLL_LOOP_START
4437- while ( decode <= end_decode )
4438- {
4439- stbir__simdfX d0,a0,d1,a1;
4440- STBIR_NO_UNROLL(decode);
4441- stbir__simdfX_load( d0, decode-2*stbir__simdfX_float_count );
4442- stbir__simdfX_load( d1, decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count );
4443- stbir__simdfX_a1a1( a0, d0, STBIR_onesX );
4444- stbir__simdfX_a1a1( a1, d1, STBIR_onesX );
4445- stbir__simdfX_mult( d0, d0, a0 );
4446- stbir__simdfX_mult( d1, d1, a1 );
4447- stbir__simdfX_store ( decode-2*stbir__simdfX_float_count, d0 );
4448- stbir__simdfX_store ( decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count, d1 );
4449- decode += 2 * stbir__simdfX_float_count;
4450- }
4451- decode -= 2 * stbir__simdfX_float_count;
4452- #endif
4453-
4454- STBIR_SIMD_NO_UNROLL_LOOP_START
4455- while( decode < end_decode )
4456- {
4457- float alpha = decode[1];
4458- STBIR_SIMD_NO_UNROLL(decode);
4459- decode[0] *= alpha;
4460- decode += 2;
4461- }
4462-}
4463-
4464-static void stbir__simple_alpha_unweight_4ch( float * encode_buffer, int width_times_channels )
4465-{
4466- float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
4467- float const * end_output = encode_buffer + width_times_channels;
4468-
4469- STBIR_SIMD_NO_UNROLL_LOOP_START
4470- do {
4471- float alpha = encode[3];
4472-
4473-#ifdef STBIR_SIMD
4474- stbir__simdf i,ia;
4475- STBIR_SIMD_NO_UNROLL(encode);
4476- if ( alpha >= stbir__small_float )
4477- {
4478- stbir__simdf_load1frep4( ia, 1.0f / alpha );
4479- stbir__simdf_load( i, encode );
4480- stbir__simdf_mult( i, i, ia );
4481- stbir__simdf_store( encode, i );
4482- encode[3] = alpha;
4483- }
4484-#else
4485- if ( alpha >= stbir__small_float )
4486- {
4487- float ialpha = 1.0f / alpha;
4488- encode[0] *= ialpha;
4489- encode[1] *= ialpha;
4490- encode[2] *= ialpha;
4491- }
4492-#endif
4493- encode += 4;
4494- } while ( encode < end_output );
4495-}
4496-
4497-static void stbir__simple_alpha_unweight_2ch( float * encode_buffer, int width_times_channels )
4498-{
4499- float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
4500- float const * end_output = encode_buffer + width_times_channels;
4501-
4502- do {
4503- float alpha = encode[1];
4504- if ( alpha >= stbir__small_float )
4505- encode[0] /= alpha;
4506- encode += 2;
4507- } while ( encode < end_output );
4508-}
4509-
4510-
4511-// only used in RGB->BGR or BGR->RGB
4512-static void stbir__simple_flip_3ch( float * decode_buffer, int width_times_channels )
4513-{
4514- float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
4515- float const * end_decode = decode_buffer + width_times_channels;
4516-
4517-#ifdef STBIR_SIMD
4518- #ifdef stbir__simdf_swiz2 // do we have two argument swizzles?
4519- end_decode -= 12;
4520- STBIR_NO_UNROLL_LOOP_START
4521- while( decode <= end_decode )
4522- {
4523- // on arm64 8 instructions, no overlapping stores
4524- stbir__simdf a,b,c,na,nb;
4525- STBIR_SIMD_NO_UNROLL(decode);
4526- stbir__simdf_load( a, decode );
4527- stbir__simdf_load( b, decode+4 );
4528- stbir__simdf_load( c, decode+8 );
4529-
4530- na = stbir__simdf_swiz2( a, b, 2, 1, 0, 5 );
4531- b = stbir__simdf_swiz2( a, b, 4, 3, 6, 7 );
4532- nb = stbir__simdf_swiz2( b, c, 0, 1, 4, 3 );
4533- c = stbir__simdf_swiz2( b, c, 2, 7, 6, 5 );
4534-
4535- stbir__simdf_store( decode, na );
4536- stbir__simdf_store( decode+4, nb );
4537- stbir__simdf_store( decode+8, c );
4538- decode += 12;
4539- }
4540- end_decode += 12;
4541- #else
4542- end_decode -= 24;
4543- STBIR_NO_UNROLL_LOOP_START
4544- while( decode <= end_decode )
4545- {
4546- // 26 instructions on x64
4547- stbir__simdf a,b,c,d,e,f,g;
4548- float i21, i23;
4549- STBIR_SIMD_NO_UNROLL(decode);
4550- stbir__simdf_load( a, decode );
4551- stbir__simdf_load( b, decode+3 );
4552- stbir__simdf_load( c, decode+6 );
4553- stbir__simdf_load( d, decode+9 );
4554- stbir__simdf_load( e, decode+12 );
4555- stbir__simdf_load( f, decode+15 );
4556- stbir__simdf_load( g, decode+18 );
4557-
4558- a = stbir__simdf_swiz( a, 2, 1, 0, 3 );
4559- b = stbir__simdf_swiz( b, 2, 1, 0, 3 );
4560- c = stbir__simdf_swiz( c, 2, 1, 0, 3 );
4561- d = stbir__simdf_swiz( d, 2, 1, 0, 3 );
4562- e = stbir__simdf_swiz( e, 2, 1, 0, 3 );
4563- f = stbir__simdf_swiz( f, 2, 1, 0, 3 );
4564- g = stbir__simdf_swiz( g, 2, 1, 0, 3 );
4565-
4566- // stores overlap, need to be in order,
4567- stbir__simdf_store( decode, a );
4568- i21 = decode[21];
4569- stbir__simdf_store( decode+3, b );
4570- i23 = decode[23];
4571- stbir__simdf_store( decode+6, c );
4572- stbir__simdf_store( decode+9, d );
4573- stbir__simdf_store( decode+12, e );
4574- stbir__simdf_store( decode+15, f );
4575- stbir__simdf_store( decode+18, g );
4576- decode[21] = i23;
4577- decode[23] = i21;
4578- decode += 24;
4579- }
4580- end_decode += 24;
4581- #endif
4582-#else
4583- end_decode -= 12;
4584- STBIR_NO_UNROLL_LOOP_START
4585- while( decode <= end_decode )
4586- {
4587- // 16 instructions
4588- float t0,t1,t2,t3;
4589- STBIR_NO_UNROLL(decode);
4590- t0 = decode[0]; t1 = decode[3]; t2 = decode[6]; t3 = decode[9];
4591- decode[0] = decode[2]; decode[3] = decode[5]; decode[6] = decode[8]; decode[9] = decode[11];
4592- decode[2] = t0; decode[5] = t1; decode[8] = t2; decode[11] = t3;
4593- decode += 12;
4594- }
4595- end_decode += 12;
4596-#endif
4597-
4598- STBIR_NO_UNROLL_LOOP_START
4599- while( decode < end_decode )
4600- {
4601- float t = decode[0];
4602- STBIR_NO_UNROLL(decode);
4603- decode[0] = decode[2];
4604- decode[2] = t;
4605- decode += 3;
4606- }
4607-}
4608-
4609-
4610-
4611-static void stbir__decode_scanline(stbir__info const * stbir_info, int n, float * output_buffer STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
4612-{
4613- int channels = stbir_info->channels;
4614- int effective_channels = stbir_info->effective_channels;
4615- int input_sample_in_bytes = stbir__type_size[stbir_info->input_type] * channels;
4616- stbir_edge edge_horizontal = stbir_info->horizontal.edge;
4617- stbir_edge edge_vertical = stbir_info->vertical.edge;
4618- int row = stbir__edge_wrap(edge_vertical, n, stbir_info->vertical.scale_info.input_full_size);
4619- const void* input_plane_data = ( (char *) stbir_info->input_data ) + (size_t)row * (size_t) stbir_info->input_stride_bytes;
4620- stbir__span const * spans = stbir_info->scanline_extents.spans;
4621- float * full_decode_buffer = output_buffer - stbir_info->scanline_extents.conservative.n0 * effective_channels;
4622- float * last_decoded = 0;
4623-
4624- // if we are on edge_zero, and we get in here with an out of bounds n, then the calculate filters has failed
4625- STBIR_ASSERT( !(edge_vertical == STBIR_EDGE_ZERO && (n < 0 || n >= stbir_info->vertical.scale_info.input_full_size)) );
4626-
4627- do
4628- {
4629- float * decode_buffer;
4630- void const * input_data;
4631- float * end_decode;
4632- int width_times_channels;
4633- int width;
4634-
4635- if ( spans->n1 < spans->n0 )
4636- break;
4637-
4638- width = spans->n1 + 1 - spans->n0;
4639- decode_buffer = full_decode_buffer + spans->n0 * effective_channels;
4640- end_decode = full_decode_buffer + ( spans->n1 + 1 ) * effective_channels;
4641- width_times_channels = width * channels;
4642-
4643- // read directly out of input plane by default
4644- input_data = ( (char*)input_plane_data ) + spans->pixel_offset_for_input * input_sample_in_bytes;
4645-
4646- // if we have an input callback, call it to get the input data
4647- if ( stbir_info->in_pixels_cb )
4648- {
4649- // call the callback with a temp buffer (that they can choose to use or not). the temp is just right aligned memory in the decode_buffer itself
4650- input_data = stbir_info->in_pixels_cb( ( (char*) end_decode ) - ( width * input_sample_in_bytes ) + ( ( stbir_info->input_type != STBIR_TYPE_FLOAT ) ? ( sizeof(float)*STBIR_INPUT_CALLBACK_PADDING ) : 0 ), input_plane_data, width, spans->pixel_offset_for_input, row, stbir_info->user_data );
4651- }
4652-
4653- STBIR_PROFILE_START( decode );
4654- // convert the pixels info the float decode_buffer, (we index from end_decode, so that when channels<effective_channels, we are right justified in the buffer)
4655- last_decoded = stbir_info->decode_pixels( (float*)end_decode - width_times_channels, width_times_channels, input_data );
4656- STBIR_PROFILE_END( decode );
4657-
4658- if (stbir_info->alpha_weight)
4659- {
4660- STBIR_PROFILE_START( alpha );
4661- stbir_info->alpha_weight( decode_buffer, width_times_channels );
4662- STBIR_PROFILE_END( alpha );
4663- }
4664-
4665- ++spans;
4666- } while ( spans <= ( &stbir_info->scanline_extents.spans[1] ) );
4667-
4668- // handle the edge_wrap filter (all other types are handled back out at the calculate_filter stage)
4669- // basically the idea here is that if we have the whole scanline in memory, we don't redecode the
4670- // wrapped edge pixels, and instead just memcpy them from the scanline into the edge positions
4671- if ( ( edge_horizontal == STBIR_EDGE_WRAP ) && ( stbir_info->scanline_extents.edge_sizes[0] | stbir_info->scanline_extents.edge_sizes[1] ) )
4672- {
4673- // this code only runs if we're in edge_wrap, and we're doing the entire scanline
4674- int e, start_x[2];
4675- int input_full_size = stbir_info->horizontal.scale_info.input_full_size;
4676-
4677- start_x[0] = -stbir_info->scanline_extents.edge_sizes[0]; // left edge start x
4678- start_x[1] = input_full_size; // right edge
4679-
4680- for( e = 0; e < 2 ; e++ )
4681- {
4682- // do each margin
4683- int margin = stbir_info->scanline_extents.edge_sizes[e];
4684- if ( margin )
4685- {
4686- int x = start_x[e];
4687- float * marg = full_decode_buffer + x * effective_channels;
4688- float const * src = full_decode_buffer + stbir__edge_wrap(edge_horizontal, x, input_full_size) * effective_channels;
4689- STBIR_MEMCPY( marg, src, margin * effective_channels * sizeof(float) );
4690- if ( e == 1 ) last_decoded = marg + margin * effective_channels;
4691- }
4692- }
4693- }
4694-
4695- // some of the horizontal gathers read one float off the edge (which is masked out), but we force a zero here to make sure no NaNs leak in
4696- // (we can't pre-zero it, because the input callback can use that area as padding)
4697- last_decoded[0] = 0.0f;
4698-
4699- // we clear this extra float, because the final output pixel filter kernel might have used one less coeff than the max filter width
4700- // when this happens, we do read that pixel from the input, so it too could be Nan, so just zero an extra one.
4701- // this fits because each scanline is padded by three floats (STBIR_INPUT_CALLBACK_PADDING)
4702- last_decoded[1] = 0.0f;
4703-}
4704-
4705-
4706-//=================
4707-// Do 1 channel horizontal routines
4708-
4709-#ifdef STBIR_SIMD
4710-
4711-#define stbir__1_coeff_only() \
4712- stbir__simdf tot,c; \
4713- STBIR_SIMD_NO_UNROLL(decode); \
4714- stbir__simdf_load1( c, hc ); \
4715- stbir__simdf_mult1_mem( tot, c, decode );
4716-
4717-#define stbir__2_coeff_only() \
4718- stbir__simdf tot,c,d; \
4719- STBIR_SIMD_NO_UNROLL(decode); \
4720- stbir__simdf_load2z( c, hc ); \
4721- stbir__simdf_load2( d, decode ); \
4722- stbir__simdf_mult( tot, c, d ); \
4723- stbir__simdf_0123to1230( c, tot ); \
4724- stbir__simdf_add1( tot, tot, c );
4725-
4726-#define stbir__3_coeff_only() \
4727- stbir__simdf tot,c,t; \
4728- STBIR_SIMD_NO_UNROLL(decode); \
4729- stbir__simdf_load( c, hc ); \
4730- stbir__simdf_mult_mem( tot, c, decode ); \
4731- stbir__simdf_0123to1230( c, tot ); \
4732- stbir__simdf_0123to2301( t, tot ); \
4733- stbir__simdf_add1( tot, tot, c ); \
4734- stbir__simdf_add1( tot, tot, t );
4735-
4736-#define stbir__store_output_tiny() \
4737- stbir__simdf_store1( output, tot ); \
4738- horizontal_coefficients += coefficient_width; \
4739- ++horizontal_contributors; \
4740- output += 1;
4741-
4742-#define stbir__4_coeff_start() \
4743- stbir__simdf tot,c; \
4744- STBIR_SIMD_NO_UNROLL(decode); \
4745- stbir__simdf_load( c, hc ); \
4746- stbir__simdf_mult_mem( tot, c, decode ); \
4747-
4748-#define stbir__4_coeff_continue_from_4( ofs ) \
4749- STBIR_SIMD_NO_UNROLL(decode); \
4750- stbir__simdf_load( c, hc + (ofs) ); \
4751- stbir__simdf_madd_mem( tot, tot, c, decode+(ofs) );
4752-
4753-#define stbir__1_coeff_remnant( ofs ) \
4754- { stbir__simdf d; \
4755- stbir__simdf_load1z( c, hc + (ofs) ); \
4756- stbir__simdf_load1( d, decode + (ofs) ); \
4757- stbir__simdf_madd( tot, tot, d, c ); }
4758-
4759-#define stbir__2_coeff_remnant( ofs ) \
4760- { stbir__simdf d; \
4761- stbir__simdf_load2z( c, hc+(ofs) ); \
4762- stbir__simdf_load2( d, decode+(ofs) ); \
4763- stbir__simdf_madd( tot, tot, d, c ); }
4764-
4765-#define stbir__3_coeff_setup() \
4766- stbir__simdf mask; \
4767- stbir__simdf_load( mask, STBIR_mask + 3 );
4768-
4769-#define stbir__3_coeff_remnant( ofs ) \
4770- stbir__simdf_load( c, hc+(ofs) ); \
4771- stbir__simdf_and( c, c, mask ); \
4772- stbir__simdf_madd_mem( tot, tot, c, decode+(ofs) );
4773-
4774-#define stbir__store_output() \
4775- stbir__simdf_0123to2301( c, tot ); \
4776- stbir__simdf_add( tot, tot, c ); \
4777- stbir__simdf_0123to1230( c, tot ); \
4778- stbir__simdf_add1( tot, tot, c ); \
4779- stbir__simdf_store1( output, tot ); \
4780- horizontal_coefficients += coefficient_width; \
4781- ++horizontal_contributors; \
4782- output += 1;
4783-
4784-#else
4785-
4786-#define stbir__1_coeff_only() \
4787- float tot; \
4788- tot = decode[0]*hc[0];
4789-
4790-#define stbir__2_coeff_only() \
4791- float tot; \
4792- tot = decode[0] * hc[0]; \
4793- tot += decode[1] * hc[1];
4794-
4795-#define stbir__3_coeff_only() \
4796- float tot; \
4797- tot = decode[0] * hc[0]; \
4798- tot += decode[1] * hc[1]; \
4799- tot += decode[2] * hc[2];
4800-
4801-#define stbir__store_output_tiny() \
4802- output[0] = tot; \
4803- horizontal_coefficients += coefficient_width; \
4804- ++horizontal_contributors; \
4805- output += 1;
4806-
4807-#define stbir__4_coeff_start() \
4808- float tot0,tot1,tot2,tot3; \
4809- tot0 = decode[0] * hc[0]; \
4810- tot1 = decode[1] * hc[1]; \
4811- tot2 = decode[2] * hc[2]; \
4812- tot3 = decode[3] * hc[3];
4813-
4814-#define stbir__4_coeff_continue_from_4( ofs ) \
4815- tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
4816- tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
4817- tot2 += decode[2+(ofs)] * hc[2+(ofs)]; \
4818- tot3 += decode[3+(ofs)] * hc[3+(ofs)];
4819-
4820-#define stbir__1_coeff_remnant( ofs ) \
4821- tot0 += decode[0+(ofs)] * hc[0+(ofs)];
4822-
4823-#define stbir__2_coeff_remnant( ofs ) \
4824- tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
4825- tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
4826-
4827-#define stbir__3_coeff_remnant( ofs ) \
4828- tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
4829- tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
4830- tot2 += decode[2+(ofs)] * hc[2+(ofs)];
4831-
4832-#define stbir__store_output() \
4833- output[0] = (tot0+tot2)+(tot1+tot3); \
4834- horizontal_coefficients += coefficient_width; \
4835- ++horizontal_contributors; \
4836- output += 1;
4837-
4838-#endif
4839-
4840-#define STBIR__horizontal_channels 1
4841-#define STB_IMAGE_RESIZE_DO_HORIZONTALS
4842-#include STBIR__HEADER_FILENAME
4843-
4844-
4845-//=================
4846-// Do 2 channel horizontal routines
4847-
4848-#ifdef STBIR_SIMD
4849-
4850-#define stbir__1_coeff_only() \
4851- stbir__simdf tot,c,d; \
4852- STBIR_SIMD_NO_UNROLL(decode); \
4853- stbir__simdf_load1z( c, hc ); \
4854- stbir__simdf_0123to0011( c, c ); \
4855- stbir__simdf_load2( d, decode ); \
4856- stbir__simdf_mult( tot, d, c );
4857-
4858-#define stbir__2_coeff_only() \
4859- stbir__simdf tot,c; \
4860- STBIR_SIMD_NO_UNROLL(decode); \
4861- stbir__simdf_load2( c, hc ); \
4862- stbir__simdf_0123to0011( c, c ); \
4863- stbir__simdf_mult_mem( tot, c, decode );
4864-
4865-#define stbir__3_coeff_only() \
4866- stbir__simdf tot,c,cs,d; \
4867- STBIR_SIMD_NO_UNROLL(decode); \
4868- stbir__simdf_load( cs, hc ); \
4869- stbir__simdf_0123to0011( c, cs ); \
4870- stbir__simdf_mult_mem( tot, c, decode ); \
4871- stbir__simdf_0123to2222( c, cs ); \
4872- stbir__simdf_load2z( d, decode+4 ); \
4873- stbir__simdf_madd( tot, tot, d, c );
4874-
4875-#define stbir__store_output_tiny() \
4876- stbir__simdf_0123to2301( c, tot ); \
4877- stbir__simdf_add( tot, tot, c ); \
4878- stbir__simdf_store2( output, tot ); \
4879- horizontal_coefficients += coefficient_width; \
4880- ++horizontal_contributors; \
4881- output += 2;
4882-
4883-#ifdef STBIR_SIMD8
4884-
4885-#define stbir__4_coeff_start() \
4886- stbir__simdf8 tot0,c,cs; \
4887- STBIR_SIMD_NO_UNROLL(decode); \
4888- stbir__simdf8_load4b( cs, hc ); \
4889- stbir__simdf8_0123to00112233( c, cs ); \
4890- stbir__simdf8_mult_mem( tot0, c, decode );
4891-
4892-#define stbir__4_coeff_continue_from_4( ofs ) \
4893- STBIR_SIMD_NO_UNROLL(decode); \
4894- stbir__simdf8_load4b( cs, hc + (ofs) ); \
4895- stbir__simdf8_0123to00112233( c, cs ); \
4896- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*2 );
4897-
4898-#define stbir__1_coeff_remnant( ofs ) \
4899- { stbir__simdf t,d; \
4900- stbir__simdf_load1z( t, hc + (ofs) ); \
4901- stbir__simdf_load2( d, decode + (ofs) * 2 ); \
4902- stbir__simdf_0123to0011( t, t ); \
4903- stbir__simdf_mult( t, t, d ); \
4904- stbir__simdf8_add4( tot0, tot0, t ); }
4905-
4906-#define stbir__2_coeff_remnant( ofs ) \
4907- { stbir__simdf t; \
4908- stbir__simdf_load2( t, hc + (ofs) ); \
4909- stbir__simdf_0123to0011( t, t ); \
4910- stbir__simdf_mult_mem( t, t, decode+(ofs)*2 ); \
4911- stbir__simdf8_add4( tot0, tot0, t ); }
4912-
4913-#define stbir__3_coeff_remnant( ofs ) \
4914- { stbir__simdf8 d; \
4915- stbir__simdf8_load4b( cs, hc + (ofs) ); \
4916- stbir__simdf8_0123to00112233( c, cs ); \
4917- stbir__simdf8_load6z( d, decode+(ofs)*2 ); \
4918- stbir__simdf8_madd( tot0, tot0, c, d ); }
4919-
4920-#define stbir__store_output() \
4921- { stbir__simdf t,d; \
4922- stbir__simdf8_add4halves( t, stbir__if_simdf8_cast_to_simdf4(tot0), tot0 ); \
4923- stbir__simdf_0123to2301( d, t ); \
4924- stbir__simdf_add( t, t, d ); \
4925- stbir__simdf_store2( output, t ); \
4926- horizontal_coefficients += coefficient_width; \
4927- ++horizontal_contributors; \
4928- output += 2; }
4929-
4930-#else
4931-
4932-#define stbir__4_coeff_start() \
4933- stbir__simdf tot0,tot1,c,cs; \
4934- STBIR_SIMD_NO_UNROLL(decode); \
4935- stbir__simdf_load( cs, hc ); \
4936- stbir__simdf_0123to0011( c, cs ); \
4937- stbir__simdf_mult_mem( tot0, c, decode ); \
4938- stbir__simdf_0123to2233( c, cs ); \
4939- stbir__simdf_mult_mem( tot1, c, decode+4 );
4940-
4941-#define stbir__4_coeff_continue_from_4( ofs ) \
4942- STBIR_SIMD_NO_UNROLL(decode); \
4943- stbir__simdf_load( cs, hc + (ofs) ); \
4944- stbir__simdf_0123to0011( c, cs ); \
4945- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 ); \
4946- stbir__simdf_0123to2233( c, cs ); \
4947- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*2+4 );
4948-
4949-#define stbir__1_coeff_remnant( ofs ) \
4950- { stbir__simdf d; \
4951- stbir__simdf_load1z( cs, hc + (ofs) ); \
4952- stbir__simdf_0123to0011( c, cs ); \
4953- stbir__simdf_load2( d, decode + (ofs) * 2 ); \
4954- stbir__simdf_madd( tot0, tot0, d, c ); }
4955-
4956-#define stbir__2_coeff_remnant( ofs ) \
4957- stbir__simdf_load2( cs, hc + (ofs) ); \
4958- stbir__simdf_0123to0011( c, cs ); \
4959- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 );
4960-
4961-#define stbir__3_coeff_remnant( ofs ) \
4962- { stbir__simdf d; \
4963- stbir__simdf_load( cs, hc + (ofs) ); \
4964- stbir__simdf_0123to0011( c, cs ); \
4965- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 ); \
4966- stbir__simdf_0123to2222( c, cs ); \
4967- stbir__simdf_load2z( d, decode + (ofs) * 2 + 4 ); \
4968- stbir__simdf_madd( tot1, tot1, d, c ); }
4969-
4970-#define stbir__store_output() \
4971- stbir__simdf_add( tot0, tot0, tot1 ); \
4972- stbir__simdf_0123to2301( c, tot0 ); \
4973- stbir__simdf_add( tot0, tot0, c ); \
4974- stbir__simdf_store2( output, tot0 ); \
4975- horizontal_coefficients += coefficient_width; \
4976- ++horizontal_contributors; \
4977- output += 2;
4978-
4979-#endif
4980-
4981-#else
4982-
4983-#define stbir__1_coeff_only() \
4984- float tota,totb,c; \
4985- c = hc[0]; \
4986- tota = decode[0]*c; \
4987- totb = decode[1]*c;
4988-
4989-#define stbir__2_coeff_only() \
4990- float tota,totb,c; \
4991- c = hc[0]; \
4992- tota = decode[0]*c; \
4993- totb = decode[1]*c; \
4994- c = hc[1]; \
4995- tota += decode[2]*c; \
4996- totb += decode[3]*c;
4997-
4998-// this weird order of add matches the simd
4999-#define stbir__3_coeff_only() \
5000- float tota,totb,c; \
5001- c = hc[0]; \
5002- tota = decode[0]*c; \
5003- totb = decode[1]*c; \
5004- c = hc[2]; \
5005- tota += decode[4]*c; \
5006- totb += decode[5]*c; \
5007- c = hc[1]; \
5008- tota += decode[2]*c; \
5009- totb += decode[3]*c;
5010-
5011-#define stbir__store_output_tiny() \
5012- output[0] = tota; \
5013- output[1] = totb; \
5014- horizontal_coefficients += coefficient_width; \
5015- ++horizontal_contributors; \
5016- output += 2;
5017-
5018-#define stbir__4_coeff_start() \
5019- float tota0,tota1,tota2,tota3,totb0,totb1,totb2,totb3,c; \
5020- c = hc[0]; \
5021- tota0 = decode[0]*c; \
5022- totb0 = decode[1]*c; \
5023- c = hc[1]; \
5024- tota1 = decode[2]*c; \
5025- totb1 = decode[3]*c; \
5026- c = hc[2]; \
5027- tota2 = decode[4]*c; \
5028- totb2 = decode[5]*c; \
5029- c = hc[3]; \
5030- tota3 = decode[6]*c; \
5031- totb3 = decode[7]*c;
5032-
5033-#define stbir__4_coeff_continue_from_4( ofs ) \
5034- c = hc[0+(ofs)]; \
5035- tota0 += decode[0+(ofs)*2]*c; \
5036- totb0 += decode[1+(ofs)*2]*c; \
5037- c = hc[1+(ofs)]; \
5038- tota1 += decode[2+(ofs)*2]*c; \
5039- totb1 += decode[3+(ofs)*2]*c; \
5040- c = hc[2+(ofs)]; \
5041- tota2 += decode[4+(ofs)*2]*c; \
5042- totb2 += decode[5+(ofs)*2]*c; \
5043- c = hc[3+(ofs)]; \
5044- tota3 += decode[6+(ofs)*2]*c; \
5045- totb3 += decode[7+(ofs)*2]*c;
5046-
5047-#define stbir__1_coeff_remnant( ofs ) \
5048- c = hc[0+(ofs)]; \
5049- tota0 += decode[0+(ofs)*2] * c; \
5050- totb0 += decode[1+(ofs)*2] * c;
5051-
5052-#define stbir__2_coeff_remnant( ofs ) \
5053- c = hc[0+(ofs)]; \
5054- tota0 += decode[0+(ofs)*2] * c; \
5055- totb0 += decode[1+(ofs)*2] * c; \
5056- c = hc[1+(ofs)]; \
5057- tota1 += decode[2+(ofs)*2] * c; \
5058- totb1 += decode[3+(ofs)*2] * c;
5059-
5060-#define stbir__3_coeff_remnant( ofs ) \
5061- c = hc[0+(ofs)]; \
5062- tota0 += decode[0+(ofs)*2] * c; \
5063- totb0 += decode[1+(ofs)*2] * c; \
5064- c = hc[1+(ofs)]; \
5065- tota1 += decode[2+(ofs)*2] * c; \
5066- totb1 += decode[3+(ofs)*2] * c; \
5067- c = hc[2+(ofs)]; \
5068- tota2 += decode[4+(ofs)*2] * c; \
5069- totb2 += decode[5+(ofs)*2] * c;
5070-
5071-#define stbir__store_output() \
5072- output[0] = (tota0+tota2)+(tota1+tota3); \
5073- output[1] = (totb0+totb2)+(totb1+totb3); \
5074- horizontal_coefficients += coefficient_width; \
5075- ++horizontal_contributors; \
5076- output += 2;
5077-
5078-#endif
5079-
5080-#define STBIR__horizontal_channels 2
5081-#define STB_IMAGE_RESIZE_DO_HORIZONTALS
5082-#include STBIR__HEADER_FILENAME
5083-
5084-
5085-//=================
5086-// Do 3 channel horizontal routines
5087-
5088-#ifdef STBIR_SIMD
5089-
5090-#define stbir__1_coeff_only() \
5091- stbir__simdf tot,c,d; \
5092- STBIR_SIMD_NO_UNROLL(decode); \
5093- stbir__simdf_load1z( c, hc ); \
5094- stbir__simdf_0123to0001( c, c ); \
5095- stbir__simdf_load( d, decode ); \
5096- stbir__simdf_mult( tot, d, c );
5097-
5098-#define stbir__2_coeff_only() \
5099- stbir__simdf tot,c,cs,d; \
5100- STBIR_SIMD_NO_UNROLL(decode); \
5101- stbir__simdf_load2( cs, hc ); \
5102- stbir__simdf_0123to0000( c, cs ); \
5103- stbir__simdf_load( d, decode ); \
5104- stbir__simdf_mult( tot, d, c ); \
5105- stbir__simdf_0123to1111( c, cs ); \
5106- stbir__simdf_load( d, decode+3 ); \
5107- stbir__simdf_madd( tot, tot, d, c );
5108-
5109-#define stbir__3_coeff_only() \
5110- stbir__simdf tot,c,d,cs; \
5111- STBIR_SIMD_NO_UNROLL(decode); \
5112- stbir__simdf_load( cs, hc ); \
5113- stbir__simdf_0123to0000( c, cs ); \
5114- stbir__simdf_load( d, decode ); \
5115- stbir__simdf_mult( tot, d, c ); \
5116- stbir__simdf_0123to1111( c, cs ); \
5117- stbir__simdf_load( d, decode+3 ); \
5118- stbir__simdf_madd( tot, tot, d, c ); \
5119- stbir__simdf_0123to2222( c, cs ); \
5120- stbir__simdf_load( d, decode+6 ); \
5121- stbir__simdf_madd( tot, tot, d, c );
5122-
5123-#define stbir__store_output_tiny() \
5124- stbir__simdf_store2( output, tot ); \
5125- stbir__simdf_0123to2301( tot, tot ); \
5126- stbir__simdf_store1( output+2, tot ); \
5127- horizontal_coefficients += coefficient_width; \
5128- ++horizontal_contributors; \
5129- output += 3;
5130-
5131-#ifdef STBIR_SIMD8
5132-
5133-// we're loading from the XXXYYY decode by -1 to get the XXXYYY into different halves of the AVX reg fyi
5134-#define stbir__4_coeff_start() \
5135- stbir__simdf8 tot0,tot1,c,cs; stbir__simdf t; \
5136- STBIR_SIMD_NO_UNROLL(decode); \
5137- stbir__simdf8_load4b( cs, hc ); \
5138- stbir__simdf8_0123to00001111( c, cs ); \
5139- stbir__simdf8_mult_mem( tot0, c, decode - 1 ); \
5140- stbir__simdf8_0123to22223333( c, cs ); \
5141- stbir__simdf8_mult_mem( tot1, c, decode+6 - 1 );
5142-
5143-#define stbir__4_coeff_continue_from_4( ofs ) \
5144- STBIR_SIMD_NO_UNROLL(decode); \
5145- stbir__simdf8_load4b( cs, hc + (ofs) ); \
5146- stbir__simdf8_0123to00001111( c, cs ); \
5147- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 ); \
5148- stbir__simdf8_0123to22223333( c, cs ); \
5149- stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*3 + 6 - 1 );
5150-
5151-#define stbir__1_coeff_remnant( ofs ) \
5152- STBIR_SIMD_NO_UNROLL(decode); \
5153- stbir__simdf_load1rep4( t, hc + (ofs) ); \
5154- stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*3 - 1 );
5155-
5156-#define stbir__2_coeff_remnant( ofs ) \
5157- STBIR_SIMD_NO_UNROLL(decode); \
5158- stbir__simdf8_load4b( cs, hc + (ofs) - 2 ); \
5159- stbir__simdf8_0123to22223333( c, cs ); \
5160- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 );
5161-
5162- #define stbir__3_coeff_remnant( ofs ) \
5163- STBIR_SIMD_NO_UNROLL(decode); \
5164- stbir__simdf8_load4b( cs, hc + (ofs) ); \
5165- stbir__simdf8_0123to00001111( c, cs ); \
5166- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 ); \
5167- stbir__simdf8_0123to2222( t, cs ); \
5168- stbir__simdf8_madd_mem4( tot1, tot1, t, decode+(ofs)*3 + 6 - 1 );
5169-
5170-#define stbir__store_output() \
5171- stbir__simdf8_add( tot0, tot0, tot1 ); \
5172- stbir__simdf_0123to1230( t, stbir__if_simdf8_cast_to_simdf4( tot0 ) ); \
5173- stbir__simdf8_add4halves( t, t, tot0 ); \
5174- horizontal_coefficients += coefficient_width; \
5175- ++horizontal_contributors; \
5176- output += 3; \
5177- if ( output < output_end ) \
5178- { \
5179- stbir__simdf_store( output-3, t ); \
5180- continue; \
5181- } \
5182- { stbir__simdf tt; stbir__simdf_0123to2301( tt, t ); \
5183- stbir__simdf_store2( output-3, t ); \
5184- stbir__simdf_store1( output+2-3, tt ); } \
5185- break;
5186-
5187-
5188-#else
5189-
5190-#define stbir__4_coeff_start() \
5191- stbir__simdf tot0,tot1,tot2,c,cs; \
5192- STBIR_SIMD_NO_UNROLL(decode); \
5193- stbir__simdf_load( cs, hc ); \
5194- stbir__simdf_0123to0001( c, cs ); \
5195- stbir__simdf_mult_mem( tot0, c, decode ); \
5196- stbir__simdf_0123to1122( c, cs ); \
5197- stbir__simdf_mult_mem( tot1, c, decode+4 ); \
5198- stbir__simdf_0123to2333( c, cs ); \
5199- stbir__simdf_mult_mem( tot2, c, decode+8 );
5200-
5201-#define stbir__4_coeff_continue_from_4( ofs ) \
5202- STBIR_SIMD_NO_UNROLL(decode); \
5203- stbir__simdf_load( cs, hc + (ofs) ); \
5204- stbir__simdf_0123to0001( c, cs ); \
5205- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
5206- stbir__simdf_0123to1122( c, cs ); \
5207- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*3+4 ); \
5208- stbir__simdf_0123to2333( c, cs ); \
5209- stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*3+8 );
5210-
5211-#define stbir__1_coeff_remnant( ofs ) \
5212- STBIR_SIMD_NO_UNROLL(decode); \
5213- stbir__simdf_load1z( c, hc + (ofs) ); \
5214- stbir__simdf_0123to0001( c, c ); \
5215- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 );
5216-
5217-#define stbir__2_coeff_remnant( ofs ) \
5218- { stbir__simdf d; \
5219- STBIR_SIMD_NO_UNROLL(decode); \
5220- stbir__simdf_load2z( cs, hc + (ofs) ); \
5221- stbir__simdf_0123to0001( c, cs ); \
5222- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
5223- stbir__simdf_0123to1122( c, cs ); \
5224- stbir__simdf_load2z( d, decode+(ofs)*3+4 ); \
5225- stbir__simdf_madd( tot1, tot1, c, d ); }
5226-
5227-#define stbir__3_coeff_remnant( ofs ) \
5228- { stbir__simdf d; \
5229- STBIR_SIMD_NO_UNROLL(decode); \
5230- stbir__simdf_load( cs, hc + (ofs) ); \
5231- stbir__simdf_0123to0001( c, cs ); \
5232- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
5233- stbir__simdf_0123to1122( c, cs ); \
5234- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*3+4 ); \
5235- stbir__simdf_0123to2222( c, cs ); \
5236- stbir__simdf_load1z( d, decode+(ofs)*3+8 ); \
5237- stbir__simdf_madd( tot2, tot2, c, d ); }
5238-
5239-#define stbir__store_output() \
5240- stbir__simdf_0123ABCDto3ABx( c, tot0, tot1 ); \
5241- stbir__simdf_0123ABCDto23Ax( cs, tot1, tot2 ); \
5242- stbir__simdf_0123to1230( tot2, tot2 ); \
5243- stbir__simdf_add( tot0, tot0, cs ); \
5244- stbir__simdf_add( c, c, tot2 ); \
5245- stbir__simdf_add( tot0, tot0, c ); \
5246- horizontal_coefficients += coefficient_width; \
5247- ++horizontal_contributors; \
5248- output += 3; \
5249- if ( output < output_end ) \
5250- { \
5251- stbir__simdf_store( output-3, tot0 ); \
5252- continue; \
5253- } \
5254- stbir__simdf_0123to2301( tot1, tot0 ); \
5255- stbir__simdf_store2( output-3, tot0 ); \
5256- stbir__simdf_store1( output+2-3, tot1 ); \
5257- break;
5258-
5259-#endif
5260-
5261-#else
5262-
5263-#define stbir__1_coeff_only() \
5264- float tot0, tot1, tot2, c; \
5265- c = hc[0]; \
5266- tot0 = decode[0]*c; \
5267- tot1 = decode[1]*c; \
5268- tot2 = decode[2]*c;
5269-
5270-#define stbir__2_coeff_only() \
5271- float tot0, tot1, tot2, c; \
5272- c = hc[0]; \
5273- tot0 = decode[0]*c; \
5274- tot1 = decode[1]*c; \
5275- tot2 = decode[2]*c; \
5276- c = hc[1]; \
5277- tot0 += decode[3]*c; \
5278- tot1 += decode[4]*c; \
5279- tot2 += decode[5]*c;
5280-
5281-#define stbir__3_coeff_only() \
5282- float tot0, tot1, tot2, c; \
5283- c = hc[0]; \
5284- tot0 = decode[0]*c; \
5285- tot1 = decode[1]*c; \
5286- tot2 = decode[2]*c; \
5287- c = hc[1]; \
5288- tot0 += decode[3]*c; \
5289- tot1 += decode[4]*c; \
5290- tot2 += decode[5]*c; \
5291- c = hc[2]; \
5292- tot0 += decode[6]*c; \
5293- tot1 += decode[7]*c; \
5294- tot2 += decode[8]*c;
5295-
5296-#define stbir__store_output_tiny() \
5297- output[0] = tot0; \
5298- output[1] = tot1; \
5299- output[2] = tot2; \
5300- horizontal_coefficients += coefficient_width; \
5301- ++horizontal_contributors; \
5302- output += 3;
5303-
5304-#define stbir__4_coeff_start() \
5305- float tota0,tota1,tota2,totb0,totb1,totb2,totc0,totc1,totc2,totd0,totd1,totd2,c; \
5306- c = hc[0]; \
5307- tota0 = decode[0]*c; \
5308- tota1 = decode[1]*c; \
5309- tota2 = decode[2]*c; \
5310- c = hc[1]; \
5311- totb0 = decode[3]*c; \
5312- totb1 = decode[4]*c; \
5313- totb2 = decode[5]*c; \
5314- c = hc[2]; \
5315- totc0 = decode[6]*c; \
5316- totc1 = decode[7]*c; \
5317- totc2 = decode[8]*c; \
5318- c = hc[3]; \
5319- totd0 = decode[9]*c; \
5320- totd1 = decode[10]*c; \
5321- totd2 = decode[11]*c;
5322-
5323-#define stbir__4_coeff_continue_from_4( ofs ) \
5324- c = hc[0+(ofs)]; \
5325- tota0 += decode[0+(ofs)*3]*c; \
5326- tota1 += decode[1+(ofs)*3]*c; \
5327- tota2 += decode[2+(ofs)*3]*c; \
5328- c = hc[1+(ofs)]; \
5329- totb0 += decode[3+(ofs)*3]*c; \
5330- totb1 += decode[4+(ofs)*3]*c; \
5331- totb2 += decode[5+(ofs)*3]*c; \
5332- c = hc[2+(ofs)]; \
5333- totc0 += decode[6+(ofs)*3]*c; \
5334- totc1 += decode[7+(ofs)*3]*c; \
5335- totc2 += decode[8+(ofs)*3]*c; \
5336- c = hc[3+(ofs)]; \
5337- totd0 += decode[9+(ofs)*3]*c; \
5338- totd1 += decode[10+(ofs)*3]*c; \
5339- totd2 += decode[11+(ofs)*3]*c;
5340-
5341-#define stbir__1_coeff_remnant( ofs ) \
5342- c = hc[0+(ofs)]; \
5343- tota0 += decode[0+(ofs)*3]*c; \
5344- tota1 += decode[1+(ofs)*3]*c; \
5345- tota2 += decode[2+(ofs)*3]*c;
5346-
5347-#define stbir__2_coeff_remnant( ofs ) \
5348- c = hc[0+(ofs)]; \
5349- tota0 += decode[0+(ofs)*3]*c; \
5350- tota1 += decode[1+(ofs)*3]*c; \
5351- tota2 += decode[2+(ofs)*3]*c; \
5352- c = hc[1+(ofs)]; \
5353- totb0 += decode[3+(ofs)*3]*c; \
5354- totb1 += decode[4+(ofs)*3]*c; \
5355- totb2 += decode[5+(ofs)*3]*c; \
5356-
5357-#define stbir__3_coeff_remnant( ofs ) \
5358- c = hc[0+(ofs)]; \
5359- tota0 += decode[0+(ofs)*3]*c; \
5360- tota1 += decode[1+(ofs)*3]*c; \
5361- tota2 += decode[2+(ofs)*3]*c; \
5362- c = hc[1+(ofs)]; \
5363- totb0 += decode[3+(ofs)*3]*c; \
5364- totb1 += decode[4+(ofs)*3]*c; \
5365- totb2 += decode[5+(ofs)*3]*c; \
5366- c = hc[2+(ofs)]; \
5367- totc0 += decode[6+(ofs)*3]*c; \
5368- totc1 += decode[7+(ofs)*3]*c; \
5369- totc2 += decode[8+(ofs)*3]*c;
5370-
5371-#define stbir__store_output() \
5372- output[0] = (tota0+totc0)+(totb0+totd0); \
5373- output[1] = (tota1+totc1)+(totb1+totd1); \
5374- output[2] = (tota2+totc2)+(totb2+totd2); \
5375- horizontal_coefficients += coefficient_width; \
5376- ++horizontal_contributors; \
5377- output += 3;
5378-
5379-#endif
5380-
5381-#define STBIR__horizontal_channels 3
5382-#define STB_IMAGE_RESIZE_DO_HORIZONTALS
5383-#include STBIR__HEADER_FILENAME
5384-
5385-//=================
5386-// Do 4 channel horizontal routines
5387-
5388-#ifdef STBIR_SIMD
5389-
5390-#define stbir__1_coeff_only() \
5391- stbir__simdf tot,c; \
5392- STBIR_SIMD_NO_UNROLL(decode); \
5393- stbir__simdf_load1( c, hc ); \
5394- stbir__simdf_0123to0000( c, c ); \
5395- stbir__simdf_mult_mem( tot, c, decode );
5396-
5397-#define stbir__2_coeff_only() \
5398- stbir__simdf tot,c,cs; \
5399- STBIR_SIMD_NO_UNROLL(decode); \
5400- stbir__simdf_load2( cs, hc ); \
5401- stbir__simdf_0123to0000( c, cs ); \
5402- stbir__simdf_mult_mem( tot, c, decode ); \
5403- stbir__simdf_0123to1111( c, cs ); \
5404- stbir__simdf_madd_mem( tot, tot, c, decode+4 );
5405-
5406-#define stbir__3_coeff_only() \
5407- stbir__simdf tot,c,cs; \
5408- STBIR_SIMD_NO_UNROLL(decode); \
5409- stbir__simdf_load( cs, hc ); \
5410- stbir__simdf_0123to0000( c, cs ); \
5411- stbir__simdf_mult_mem( tot, c, decode ); \
5412- stbir__simdf_0123to1111( c, cs ); \
5413- stbir__simdf_madd_mem( tot, tot, c, decode+4 ); \
5414- stbir__simdf_0123to2222( c, cs ); \
5415- stbir__simdf_madd_mem( tot, tot, c, decode+8 );
5416-
5417-#define stbir__store_output_tiny() \
5418- stbir__simdf_store( output, tot ); \
5419- horizontal_coefficients += coefficient_width; \
5420- ++horizontal_contributors; \
5421- output += 4;
5422-
5423-#ifdef STBIR_SIMD8
5424-
5425-#define stbir__4_coeff_start() \
5426- stbir__simdf8 tot0,c,cs; stbir__simdf t; \
5427- STBIR_SIMD_NO_UNROLL(decode); \
5428- stbir__simdf8_load4b( cs, hc ); \
5429- stbir__simdf8_0123to00001111( c, cs ); \
5430- stbir__simdf8_mult_mem( tot0, c, decode ); \
5431- stbir__simdf8_0123to22223333( c, cs ); \
5432- stbir__simdf8_madd_mem( tot0, tot0, c, decode+8 );
5433-
5434-#define stbir__4_coeff_continue_from_4( ofs ) \
5435- STBIR_SIMD_NO_UNROLL(decode); \
5436- stbir__simdf8_load4b( cs, hc + (ofs) ); \
5437- stbir__simdf8_0123to00001111( c, cs ); \
5438- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
5439- stbir__simdf8_0123to22223333( c, cs ); \
5440- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 );
5441-
5442-#define stbir__1_coeff_remnant( ofs ) \
5443- STBIR_SIMD_NO_UNROLL(decode); \
5444- stbir__simdf_load1rep4( t, hc + (ofs) ); \
5445- stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*4 );
5446-
5447-#define stbir__2_coeff_remnant( ofs ) \
5448- STBIR_SIMD_NO_UNROLL(decode); \
5449- stbir__simdf8_load4b( cs, hc + (ofs) - 2 ); \
5450- stbir__simdf8_0123to22223333( c, cs ); \
5451- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 );
5452-
5453- #define stbir__3_coeff_remnant( ofs ) \
5454- STBIR_SIMD_NO_UNROLL(decode); \
5455- stbir__simdf8_load4b( cs, hc + (ofs) ); \
5456- stbir__simdf8_0123to00001111( c, cs ); \
5457- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
5458- stbir__simdf8_0123to2222( t, cs ); \
5459- stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*4+8 );
5460-
5461-#define stbir__store_output() \
5462- stbir__simdf8_add4halves( t, stbir__if_simdf8_cast_to_simdf4(tot0), tot0 ); \
5463- stbir__simdf_store( output, t ); \
5464- horizontal_coefficients += coefficient_width; \
5465- ++horizontal_contributors; \
5466- output += 4;
5467-
5468-#else
5469-
5470-#define stbir__4_coeff_start() \
5471- stbir__simdf tot0,tot1,c,cs; \
5472- STBIR_SIMD_NO_UNROLL(decode); \
5473- stbir__simdf_load( cs, hc ); \
5474- stbir__simdf_0123to0000( c, cs ); \
5475- stbir__simdf_mult_mem( tot0, c, decode ); \
5476- stbir__simdf_0123to1111( c, cs ); \
5477- stbir__simdf_mult_mem( tot1, c, decode+4 ); \
5478- stbir__simdf_0123to2222( c, cs ); \
5479- stbir__simdf_madd_mem( tot0, tot0, c, decode+8 ); \
5480- stbir__simdf_0123to3333( c, cs ); \
5481- stbir__simdf_madd_mem( tot1, tot1, c, decode+12 );
5482-
5483-#define stbir__4_coeff_continue_from_4( ofs ) \
5484- STBIR_SIMD_NO_UNROLL(decode); \
5485- stbir__simdf_load( cs, hc + (ofs) ); \
5486- stbir__simdf_0123to0000( c, cs ); \
5487- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
5488- stbir__simdf_0123to1111( c, cs ); \
5489- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 ); \
5490- stbir__simdf_0123to2222( c, cs ); \
5491- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 ); \
5492- stbir__simdf_0123to3333( c, cs ); \
5493- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+12 );
5494-
5495-#define stbir__1_coeff_remnant( ofs ) \
5496- STBIR_SIMD_NO_UNROLL(decode); \
5497- stbir__simdf_load1( c, hc + (ofs) ); \
5498- stbir__simdf_0123to0000( c, c ); \
5499- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 );
5500-
5501-#define stbir__2_coeff_remnant( ofs ) \
5502- STBIR_SIMD_NO_UNROLL(decode); \
5503- stbir__simdf_load2( cs, hc + (ofs) ); \
5504- stbir__simdf_0123to0000( c, cs ); \
5505- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
5506- stbir__simdf_0123to1111( c, cs ); \
5507- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 );
5508-
5509-#define stbir__3_coeff_remnant( ofs ) \
5510- STBIR_SIMD_NO_UNROLL(decode); \
5511- stbir__simdf_load( cs, hc + (ofs) ); \
5512- stbir__simdf_0123to0000( c, cs ); \
5513- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
5514- stbir__simdf_0123to1111( c, cs ); \
5515- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 ); \
5516- stbir__simdf_0123to2222( c, cs ); \
5517- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 );
5518-
5519-#define stbir__store_output() \
5520- stbir__simdf_add( tot0, tot0, tot1 ); \
5521- stbir__simdf_store( output, tot0 ); \
5522- horizontal_coefficients += coefficient_width; \
5523- ++horizontal_contributors; \
5524- output += 4;
5525-
5526-#endif
5527-
5528-#else
5529-
5530-#define stbir__1_coeff_only() \
5531- float p0,p1,p2,p3,c; \
5532- STBIR_SIMD_NO_UNROLL(decode); \
5533- c = hc[0]; \
5534- p0 = decode[0] * c; \
5535- p1 = decode[1] * c; \
5536- p2 = decode[2] * c; \
5537- p3 = decode[3] * c;
5538-
5539-#define stbir__2_coeff_only() \
5540- float p0,p1,p2,p3,c; \
5541- STBIR_SIMD_NO_UNROLL(decode); \
5542- c = hc[0]; \
5543- p0 = decode[0] * c; \
5544- p1 = decode[1] * c; \
5545- p2 = decode[2] * c; \
5546- p3 = decode[3] * c; \
5547- c = hc[1]; \
5548- p0 += decode[4] * c; \
5549- p1 += decode[5] * c; \
5550- p2 += decode[6] * c; \
5551- p3 += decode[7] * c;
5552-
5553-#define stbir__3_coeff_only() \
5554- float p0,p1,p2,p3,c; \
5555- STBIR_SIMD_NO_UNROLL(decode); \
5556- c = hc[0]; \
5557- p0 = decode[0] * c; \
5558- p1 = decode[1] * c; \
5559- p2 = decode[2] * c; \
5560- p3 = decode[3] * c; \
5561- c = hc[1]; \
5562- p0 += decode[4] * c; \
5563- p1 += decode[5] * c; \
5564- p2 += decode[6] * c; \
5565- p3 += decode[7] * c; \
5566- c = hc[2]; \
5567- p0 += decode[8] * c; \
5568- p1 += decode[9] * c; \
5569- p2 += decode[10] * c; \
5570- p3 += decode[11] * c;
5571-
5572-#define stbir__store_output_tiny() \
5573- output[0] = p0; \
5574- output[1] = p1; \
5575- output[2] = p2; \
5576- output[3] = p3; \
5577- horizontal_coefficients += coefficient_width; \
5578- ++horizontal_contributors; \
5579- output += 4;
5580-
5581-#define stbir__4_coeff_start() \
5582- float x0,x1,x2,x3,y0,y1,y2,y3,c; \
5583- STBIR_SIMD_NO_UNROLL(decode); \
5584- c = hc[0]; \
5585- x0 = decode[0] * c; \
5586- x1 = decode[1] * c; \
5587- x2 = decode[2] * c; \
5588- x3 = decode[3] * c; \
5589- c = hc[1]; \
5590- y0 = decode[4] * c; \
5591- y1 = decode[5] * c; \
5592- y2 = decode[6] * c; \
5593- y3 = decode[7] * c; \
5594- c = hc[2]; \
5595- x0 += decode[8] * c; \
5596- x1 += decode[9] * c; \
5597- x2 += decode[10] * c; \
5598- x3 += decode[11] * c; \
5599- c = hc[3]; \
5600- y0 += decode[12] * c; \
5601- y1 += decode[13] * c; \
5602- y2 += decode[14] * c; \
5603- y3 += decode[15] * c;
5604-
5605-#define stbir__4_coeff_continue_from_4( ofs ) \
5606- STBIR_SIMD_NO_UNROLL(decode); \
5607- c = hc[0+(ofs)]; \
5608- x0 += decode[0+(ofs)*4] * c; \
5609- x1 += decode[1+(ofs)*4] * c; \
5610- x2 += decode[2+(ofs)*4] * c; \
5611- x3 += decode[3+(ofs)*4] * c; \
5612- c = hc[1+(ofs)]; \
5613- y0 += decode[4+(ofs)*4] * c; \
5614- y1 += decode[5+(ofs)*4] * c; \
5615- y2 += decode[6+(ofs)*4] * c; \
5616- y3 += decode[7+(ofs)*4] * c; \
5617- c = hc[2+(ofs)]; \
5618- x0 += decode[8+(ofs)*4] * c; \
5619- x1 += decode[9+(ofs)*4] * c; \
5620- x2 += decode[10+(ofs)*4] * c; \
5621- x3 += decode[11+(ofs)*4] * c; \
5622- c = hc[3+(ofs)]; \
5623- y0 += decode[12+(ofs)*4] * c; \
5624- y1 += decode[13+(ofs)*4] * c; \
5625- y2 += decode[14+(ofs)*4] * c; \
5626- y3 += decode[15+(ofs)*4] * c;
5627-
5628-#define stbir__1_coeff_remnant( ofs ) \
5629- STBIR_SIMD_NO_UNROLL(decode); \
5630- c = hc[0+(ofs)]; \
5631- x0 += decode[0+(ofs)*4] * c; \
5632- x1 += decode[1+(ofs)*4] * c; \
5633- x2 += decode[2+(ofs)*4] * c; \
5634- x3 += decode[3+(ofs)*4] * c;
5635-
5636-#define stbir__2_coeff_remnant( ofs ) \
5637- STBIR_SIMD_NO_UNROLL(decode); \
5638- c = hc[0+(ofs)]; \
5639- x0 += decode[0+(ofs)*4] * c; \
5640- x1 += decode[1+(ofs)*4] * c; \
5641- x2 += decode[2+(ofs)*4] * c; \
5642- x3 += decode[3+(ofs)*4] * c; \
5643- c = hc[1+(ofs)]; \
5644- y0 += decode[4+(ofs)*4] * c; \
5645- y1 += decode[5+(ofs)*4] * c; \
5646- y2 += decode[6+(ofs)*4] * c; \
5647- y3 += decode[7+(ofs)*4] * c;
5648-
5649-#define stbir__3_coeff_remnant( ofs ) \
5650- STBIR_SIMD_NO_UNROLL(decode); \
5651- c = hc[0+(ofs)]; \
5652- x0 += decode[0+(ofs)*4] * c; \
5653- x1 += decode[1+(ofs)*4] * c; \
5654- x2 += decode[2+(ofs)*4] * c; \
5655- x3 += decode[3+(ofs)*4] * c; \
5656- c = hc[1+(ofs)]; \
5657- y0 += decode[4+(ofs)*4] * c; \
5658- y1 += decode[5+(ofs)*4] * c; \
5659- y2 += decode[6+(ofs)*4] * c; \
5660- y3 += decode[7+(ofs)*4] * c; \
5661- c = hc[2+(ofs)]; \
5662- x0 += decode[8+(ofs)*4] * c; \
5663- x1 += decode[9+(ofs)*4] * c; \
5664- x2 += decode[10+(ofs)*4] * c; \
5665- x3 += decode[11+(ofs)*4] * c;
5666-
5667-#define stbir__store_output() \
5668- output[0] = x0 + y0; \
5669- output[1] = x1 + y1; \
5670- output[2] = x2 + y2; \
5671- output[3] = x3 + y3; \
5672- horizontal_coefficients += coefficient_width; \
5673- ++horizontal_contributors; \
5674- output += 4;
5675-
5676-#endif
5677-
5678-#define STBIR__horizontal_channels 4
5679-#define STB_IMAGE_RESIZE_DO_HORIZONTALS
5680-#include STBIR__HEADER_FILENAME
5681-
5682-
5683-
5684-//=================
5685-// Do 7 channel horizontal routines
5686-
5687-#ifdef STBIR_SIMD
5688-
5689-#define stbir__1_coeff_only() \
5690- stbir__simdf tot0,tot1,c; \
5691- STBIR_SIMD_NO_UNROLL(decode); \
5692- stbir__simdf_load1( c, hc ); \
5693- stbir__simdf_0123to0000( c, c ); \
5694- stbir__simdf_mult_mem( tot0, c, decode ); \
5695- stbir__simdf_mult_mem( tot1, c, decode+3 );
5696-
5697-#define stbir__2_coeff_only() \
5698- stbir__simdf tot0,tot1,c,cs; \
5699- STBIR_SIMD_NO_UNROLL(decode); \
5700- stbir__simdf_load2( cs, hc ); \
5701- stbir__simdf_0123to0000( c, cs ); \
5702- stbir__simdf_mult_mem( tot0, c, decode ); \
5703- stbir__simdf_mult_mem( tot1, c, decode+3 ); \
5704- stbir__simdf_0123to1111( c, cs ); \
5705- stbir__simdf_madd_mem( tot0, tot0, c, decode+7 ); \
5706- stbir__simdf_madd_mem( tot1, tot1, c,decode+10 );
5707-
5708-#define stbir__3_coeff_only() \
5709- stbir__simdf tot0,tot1,c,cs; \
5710- STBIR_SIMD_NO_UNROLL(decode); \
5711- stbir__simdf_load( cs, hc ); \
5712- stbir__simdf_0123to0000( c, cs ); \
5713- stbir__simdf_mult_mem( tot0, c, decode ); \
5714- stbir__simdf_mult_mem( tot1, c, decode+3 ); \
5715- stbir__simdf_0123to1111( c, cs ); \
5716- stbir__simdf_madd_mem( tot0, tot0, c, decode+7 ); \
5717- stbir__simdf_madd_mem( tot1, tot1, c, decode+10 ); \
5718- stbir__simdf_0123to2222( c, cs ); \
5719- stbir__simdf_madd_mem( tot0, tot0, c, decode+14 ); \
5720- stbir__simdf_madd_mem( tot1, tot1, c, decode+17 );
5721-
5722-#define stbir__store_output_tiny() \
5723- stbir__simdf_store( output+3, tot1 ); \
5724- stbir__simdf_store( output, tot0 ); \
5725- horizontal_coefficients += coefficient_width; \
5726- ++horizontal_contributors; \
5727- output += 7;
5728-
5729-#ifdef STBIR_SIMD8
5730-
5731-#define stbir__4_coeff_start() \
5732- stbir__simdf8 tot0,tot1,c,cs; \
5733- STBIR_SIMD_NO_UNROLL(decode); \
5734- stbir__simdf8_load4b( cs, hc ); \
5735- stbir__simdf8_0123to00000000( c, cs ); \
5736- stbir__simdf8_mult_mem( tot0, c, decode ); \
5737- stbir__simdf8_0123to11111111( c, cs ); \
5738- stbir__simdf8_mult_mem( tot1, c, decode+7 ); \
5739- stbir__simdf8_0123to22222222( c, cs ); \
5740- stbir__simdf8_madd_mem( tot0, tot0, c, decode+14 ); \
5741- stbir__simdf8_0123to33333333( c, cs ); \
5742- stbir__simdf8_madd_mem( tot1, tot1, c, decode+21 );
5743-
5744-#define stbir__4_coeff_continue_from_4( ofs ) \
5745- STBIR_SIMD_NO_UNROLL(decode); \
5746- stbir__simdf8_load4b( cs, hc + (ofs) ); \
5747- stbir__simdf8_0123to00000000( c, cs ); \
5748- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5749- stbir__simdf8_0123to11111111( c, cs ); \
5750- stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 ); \
5751- stbir__simdf8_0123to22222222( c, cs ); \
5752- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
5753- stbir__simdf8_0123to33333333( c, cs ); \
5754- stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+21 );
5755-
5756-#define stbir__1_coeff_remnant( ofs ) \
5757- STBIR_SIMD_NO_UNROLL(decode); \
5758- stbir__simdf8_load1b( c, hc + (ofs) ); \
5759- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 );
5760-
5761-#define stbir__2_coeff_remnant( ofs ) \
5762- STBIR_SIMD_NO_UNROLL(decode); \
5763- stbir__simdf8_load1b( c, hc + (ofs) ); \
5764- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5765- stbir__simdf8_load1b( c, hc + (ofs)+1 ); \
5766- stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 );
5767-
5768-#define stbir__3_coeff_remnant( ofs ) \
5769- STBIR_SIMD_NO_UNROLL(decode); \
5770- stbir__simdf8_load4b( cs, hc + (ofs) ); \
5771- stbir__simdf8_0123to00000000( c, cs ); \
5772- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5773- stbir__simdf8_0123to11111111( c, cs ); \
5774- stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 ); \
5775- stbir__simdf8_0123to22222222( c, cs ); \
5776- stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 );
5777-
5778-#define stbir__store_output() \
5779- stbir__simdf8_add( tot0, tot0, tot1 ); \
5780- horizontal_coefficients += coefficient_width; \
5781- ++horizontal_contributors; \
5782- output += 7; \
5783- if ( output < output_end ) \
5784- { \
5785- stbir__simdf8_store( output-7, tot0 ); \
5786- continue; \
5787- } \
5788- stbir__simdf_store( output-7+3, stbir__simdf_swiz(stbir__simdf8_gettop4(tot0),0,0,1,2) ); \
5789- stbir__simdf_store( output-7, stbir__if_simdf8_cast_to_simdf4(tot0) ); \
5790- break;
5791-
5792-#else
5793-
5794-#define stbir__4_coeff_start() \
5795- stbir__simdf tot0,tot1,tot2,tot3,c,cs; \
5796- STBIR_SIMD_NO_UNROLL(decode); \
5797- stbir__simdf_load( cs, hc ); \
5798- stbir__simdf_0123to0000( c, cs ); \
5799- stbir__simdf_mult_mem( tot0, c, decode ); \
5800- stbir__simdf_mult_mem( tot1, c, decode+3 ); \
5801- stbir__simdf_0123to1111( c, cs ); \
5802- stbir__simdf_mult_mem( tot2, c, decode+7 ); \
5803- stbir__simdf_mult_mem( tot3, c, decode+10 ); \
5804- stbir__simdf_0123to2222( c, cs ); \
5805- stbir__simdf_madd_mem( tot0, tot0, c, decode+14 ); \
5806- stbir__simdf_madd_mem( tot1, tot1, c, decode+17 ); \
5807- stbir__simdf_0123to3333( c, cs ); \
5808- stbir__simdf_madd_mem( tot2, tot2, c, decode+21 ); \
5809- stbir__simdf_madd_mem( tot3, tot3, c, decode+24 );
5810-
5811-#define stbir__4_coeff_continue_from_4( ofs ) \
5812- STBIR_SIMD_NO_UNROLL(decode); \
5813- stbir__simdf_load( cs, hc + (ofs) ); \
5814- stbir__simdf_0123to0000( c, cs ); \
5815- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5816- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
5817- stbir__simdf_0123to1111( c, cs ); \
5818- stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
5819- stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 ); \
5820- stbir__simdf_0123to2222( c, cs ); \
5821- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
5822- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+17 ); \
5823- stbir__simdf_0123to3333( c, cs ); \
5824- stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+21 ); \
5825- stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+24 );
5826-
5827-#define stbir__1_coeff_remnant( ofs ) \
5828- STBIR_SIMD_NO_UNROLL(decode); \
5829- stbir__simdf_load1( c, hc + (ofs) ); \
5830- stbir__simdf_0123to0000( c, c ); \
5831- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5832- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
5833-
5834-#define stbir__2_coeff_remnant( ofs ) \
5835- STBIR_SIMD_NO_UNROLL(decode); \
5836- stbir__simdf_load2( cs, hc + (ofs) ); \
5837- stbir__simdf_0123to0000( c, cs ); \
5838- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5839- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
5840- stbir__simdf_0123to1111( c, cs ); \
5841- stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
5842- stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 );
5843-
5844-#define stbir__3_coeff_remnant( ofs ) \
5845- STBIR_SIMD_NO_UNROLL(decode); \
5846- stbir__simdf_load( cs, hc + (ofs) ); \
5847- stbir__simdf_0123to0000( c, cs ); \
5848- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5849- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
5850- stbir__simdf_0123to1111( c, cs ); \
5851- stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
5852- stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 ); \
5853- stbir__simdf_0123to2222( c, cs ); \
5854- stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
5855- stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+17 );
5856-
5857-#define stbir__store_output() \
5858- stbir__simdf_add( tot0, tot0, tot2 ); \
5859- stbir__simdf_add( tot1, tot1, tot3 ); \
5860- stbir__simdf_store( output+3, tot1 ); \
5861- stbir__simdf_store( output, tot0 ); \
5862- horizontal_coefficients += coefficient_width; \
5863- ++horizontal_contributors; \
5864- output += 7;
5865-
5866-#endif
5867-
5868-#else
5869-
5870-#define stbir__1_coeff_only() \
5871- float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
5872- c = hc[0]; \
5873- tot0 = decode[0]*c; \
5874- tot1 = decode[1]*c; \
5875- tot2 = decode[2]*c; \
5876- tot3 = decode[3]*c; \
5877- tot4 = decode[4]*c; \
5878- tot5 = decode[5]*c; \
5879- tot6 = decode[6]*c;
5880-
5881-#define stbir__2_coeff_only() \
5882- float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
5883- c = hc[0]; \
5884- tot0 = decode[0]*c; \
5885- tot1 = decode[1]*c; \
5886- tot2 = decode[2]*c; \
5887- tot3 = decode[3]*c; \
5888- tot4 = decode[4]*c; \
5889- tot5 = decode[5]*c; \
5890- tot6 = decode[6]*c; \
5891- c = hc[1]; \
5892- tot0 += decode[7]*c; \
5893- tot1 += decode[8]*c; \
5894- tot2 += decode[9]*c; \
5895- tot3 += decode[10]*c; \
5896- tot4 += decode[11]*c; \
5897- tot5 += decode[12]*c; \
5898- tot6 += decode[13]*c; \
5899-
5900-#define stbir__3_coeff_only() \
5901- float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
5902- c = hc[0]; \
5903- tot0 = decode[0]*c; \
5904- tot1 = decode[1]*c; \
5905- tot2 = decode[2]*c; \
5906- tot3 = decode[3]*c; \
5907- tot4 = decode[4]*c; \
5908- tot5 = decode[5]*c; \
5909- tot6 = decode[6]*c; \
5910- c = hc[1]; \
5911- tot0 += decode[7]*c; \
5912- tot1 += decode[8]*c; \
5913- tot2 += decode[9]*c; \
5914- tot3 += decode[10]*c; \
5915- tot4 += decode[11]*c; \
5916- tot5 += decode[12]*c; \
5917- tot6 += decode[13]*c; \
5918- c = hc[2]; \
5919- tot0 += decode[14]*c; \
5920- tot1 += decode[15]*c; \
5921- tot2 += decode[16]*c; \
5922- tot3 += decode[17]*c; \
5923- tot4 += decode[18]*c; \
5924- tot5 += decode[19]*c; \
5925- tot6 += decode[20]*c; \
5926-
5927-#define stbir__store_output_tiny() \
5928- output[0] = tot0; \
5929- output[1] = tot1; \
5930- output[2] = tot2; \
5931- output[3] = tot3; \
5932- output[4] = tot4; \
5933- output[5] = tot5; \
5934- output[6] = tot6; \
5935- horizontal_coefficients += coefficient_width; \
5936- ++horizontal_contributors; \
5937- output += 7;
5938-
5939-#define stbir__4_coeff_start() \
5940- float x0,x1,x2,x3,x4,x5,x6,y0,y1,y2,y3,y4,y5,y6,c; \
5941- STBIR_SIMD_NO_UNROLL(decode); \
5942- c = hc[0]; \
5943- x0 = decode[0] * c; \
5944- x1 = decode[1] * c; \
5945- x2 = decode[2] * c; \
5946- x3 = decode[3] * c; \
5947- x4 = decode[4] * c; \
5948- x5 = decode[5] * c; \
5949- x6 = decode[6] * c; \
5950- c = hc[1]; \
5951- y0 = decode[7] * c; \
5952- y1 = decode[8] * c; \
5953- y2 = decode[9] * c; \
5954- y3 = decode[10] * c; \
5955- y4 = decode[11] * c; \
5956- y5 = decode[12] * c; \
5957- y6 = decode[13] * c; \
5958- c = hc[2]; \
5959- x0 += decode[14] * c; \
5960- x1 += decode[15] * c; \
5961- x2 += decode[16] * c; \
5962- x3 += decode[17] * c; \
5963- x4 += decode[18] * c; \
5964- x5 += decode[19] * c; \
5965- x6 += decode[20] * c; \
5966- c = hc[3]; \
5967- y0 += decode[21] * c; \
5968- y1 += decode[22] * c; \
5969- y2 += decode[23] * c; \
5970- y3 += decode[24] * c; \
5971- y4 += decode[25] * c; \
5972- y5 += decode[26] * c; \
5973- y6 += decode[27] * c;
5974-
5975-#define stbir__4_coeff_continue_from_4( ofs ) \
5976- STBIR_SIMD_NO_UNROLL(decode); \
5977- c = hc[0+(ofs)]; \
5978- x0 += decode[0+(ofs)*7] * c; \
5979- x1 += decode[1+(ofs)*7] * c; \
5980- x2 += decode[2+(ofs)*7] * c; \
5981- x3 += decode[3+(ofs)*7] * c; \
5982- x4 += decode[4+(ofs)*7] * c; \
5983- x5 += decode[5+(ofs)*7] * c; \
5984- x6 += decode[6+(ofs)*7] * c; \
5985- c = hc[1+(ofs)]; \
5986- y0 += decode[7+(ofs)*7] * c; \
5987- y1 += decode[8+(ofs)*7] * c; \
5988- y2 += decode[9+(ofs)*7] * c; \
5989- y3 += decode[10+(ofs)*7] * c; \
5990- y4 += decode[11+(ofs)*7] * c; \
5991- y5 += decode[12+(ofs)*7] * c; \
5992- y6 += decode[13+(ofs)*7] * c; \
5993- c = hc[2+(ofs)]; \
5994- x0 += decode[14+(ofs)*7] * c; \
5995- x1 += decode[15+(ofs)*7] * c; \
5996- x2 += decode[16+(ofs)*7] * c; \
5997- x3 += decode[17+(ofs)*7] * c; \
5998- x4 += decode[18+(ofs)*7] * c; \
5999- x5 += decode[19+(ofs)*7] * c; \
6000- x6 += decode[20+(ofs)*7] * c; \
6001- c = hc[3+(ofs)]; \
6002- y0 += decode[21+(ofs)*7] * c; \
6003- y1 += decode[22+(ofs)*7] * c; \
6004- y2 += decode[23+(ofs)*7] * c; \
6005- y3 += decode[24+(ofs)*7] * c; \
6006- y4 += decode[25+(ofs)*7] * c; \
6007- y5 += decode[26+(ofs)*7] * c; \
6008- y6 += decode[27+(ofs)*7] * c;
6009-
6010-#define stbir__1_coeff_remnant( ofs ) \
6011- STBIR_SIMD_NO_UNROLL(decode); \
6012- c = hc[0+(ofs)]; \
6013- x0 += decode[0+(ofs)*7] * c; \
6014- x1 += decode[1+(ofs)*7] * c; \
6015- x2 += decode[2+(ofs)*7] * c; \
6016- x3 += decode[3+(ofs)*7] * c; \
6017- x4 += decode[4+(ofs)*7] * c; \
6018- x5 += decode[5+(ofs)*7] * c; \
6019- x6 += decode[6+(ofs)*7] * c; \
6020-
6021-#define stbir__2_coeff_remnant( ofs ) \
6022- STBIR_SIMD_NO_UNROLL(decode); \
6023- c = hc[0+(ofs)]; \
6024- x0 += decode[0+(ofs)*7] * c; \
6025- x1 += decode[1+(ofs)*7] * c; \
6026- x2 += decode[2+(ofs)*7] * c; \
6027- x3 += decode[3+(ofs)*7] * c; \
6028- x4 += decode[4+(ofs)*7] * c; \
6029- x5 += decode[5+(ofs)*7] * c; \
6030- x6 += decode[6+(ofs)*7] * c; \
6031- c = hc[1+(ofs)]; \
6032- y0 += decode[7+(ofs)*7] * c; \
6033- y1 += decode[8+(ofs)*7] * c; \
6034- y2 += decode[9+(ofs)*7] * c; \
6035- y3 += decode[10+(ofs)*7] * c; \
6036- y4 += decode[11+(ofs)*7] * c; \
6037- y5 += decode[12+(ofs)*7] * c; \
6038- y6 += decode[13+(ofs)*7] * c; \
6039-
6040-#define stbir__3_coeff_remnant( ofs ) \
6041- STBIR_SIMD_NO_UNROLL(decode); \
6042- c = hc[0+(ofs)]; \
6043- x0 += decode[0+(ofs)*7] * c; \
6044- x1 += decode[1+(ofs)*7] * c; \
6045- x2 += decode[2+(ofs)*7] * c; \
6046- x3 += decode[3+(ofs)*7] * c; \
6047- x4 += decode[4+(ofs)*7] * c; \
6048- x5 += decode[5+(ofs)*7] * c; \
6049- x6 += decode[6+(ofs)*7] * c; \
6050- c = hc[1+(ofs)]; \
6051- y0 += decode[7+(ofs)*7] * c; \
6052- y1 += decode[8+(ofs)*7] * c; \
6053- y2 += decode[9+(ofs)*7] * c; \
6054- y3 += decode[10+(ofs)*7] * c; \
6055- y4 += decode[11+(ofs)*7] * c; \
6056- y5 += decode[12+(ofs)*7] * c; \
6057- y6 += decode[13+(ofs)*7] * c; \
6058- c = hc[2+(ofs)]; \
6059- x0 += decode[14+(ofs)*7] * c; \
6060- x1 += decode[15+(ofs)*7] * c; \
6061- x2 += decode[16+(ofs)*7] * c; \
6062- x3 += decode[17+(ofs)*7] * c; \
6063- x4 += decode[18+(ofs)*7] * c; \
6064- x5 += decode[19+(ofs)*7] * c; \
6065- x6 += decode[20+(ofs)*7] * c; \
6066-
6067-#define stbir__store_output() \
6068- output[0] = x0 + y0; \
6069- output[1] = x1 + y1; \
6070- output[2] = x2 + y2; \
6071- output[3] = x3 + y3; \
6072- output[4] = x4 + y4; \
6073- output[5] = x5 + y5; \
6074- output[6] = x6 + y6; \
6075- horizontal_coefficients += coefficient_width; \
6076- ++horizontal_contributors; \
6077- output += 7;
6078-
6079-#endif
6080-
6081-#define STBIR__horizontal_channels 7
6082-#define STB_IMAGE_RESIZE_DO_HORIZONTALS
6083-#include STBIR__HEADER_FILENAME
6084-
6085-
6086-// include all of the vertical resamplers (both scatter and gather versions)
6087-
6088-#define STBIR__vertical_channels 1
6089-#define STB_IMAGE_RESIZE_DO_VERTICALS
6090-#include STBIR__HEADER_FILENAME
6091-
6092-#define STBIR__vertical_channels 1
6093-#define STB_IMAGE_RESIZE_DO_VERTICALS
6094-#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6095-#include STBIR__HEADER_FILENAME
6096-
6097-#define STBIR__vertical_channels 2
6098-#define STB_IMAGE_RESIZE_DO_VERTICALS
6099-#include STBIR__HEADER_FILENAME
6100-
6101-#define STBIR__vertical_channels 2
6102-#define STB_IMAGE_RESIZE_DO_VERTICALS
6103-#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6104-#include STBIR__HEADER_FILENAME
6105-
6106-#define STBIR__vertical_channels 3
6107-#define STB_IMAGE_RESIZE_DO_VERTICALS
6108-#include STBIR__HEADER_FILENAME
6109-
6110-#define STBIR__vertical_channels 3
6111-#define STB_IMAGE_RESIZE_DO_VERTICALS
6112-#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6113-#include STBIR__HEADER_FILENAME
6114-
6115-#define STBIR__vertical_channels 4
6116-#define STB_IMAGE_RESIZE_DO_VERTICALS
6117-#include STBIR__HEADER_FILENAME
6118-
6119-#define STBIR__vertical_channels 4
6120-#define STB_IMAGE_RESIZE_DO_VERTICALS
6121-#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6122-#include STBIR__HEADER_FILENAME
6123-
6124-#define STBIR__vertical_channels 5
6125-#define STB_IMAGE_RESIZE_DO_VERTICALS
6126-#include STBIR__HEADER_FILENAME
6127-
6128-#define STBIR__vertical_channels 5
6129-#define STB_IMAGE_RESIZE_DO_VERTICALS
6130-#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6131-#include STBIR__HEADER_FILENAME
6132-
6133-#define STBIR__vertical_channels 6
6134-#define STB_IMAGE_RESIZE_DO_VERTICALS
6135-#include STBIR__HEADER_FILENAME
6136-
6137-#define STBIR__vertical_channels 6
6138-#define STB_IMAGE_RESIZE_DO_VERTICALS
6139-#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6140-#include STBIR__HEADER_FILENAME
6141-
6142-#define STBIR__vertical_channels 7
6143-#define STB_IMAGE_RESIZE_DO_VERTICALS
6144-#include STBIR__HEADER_FILENAME
6145-
6146-#define STBIR__vertical_channels 7
6147-#define STB_IMAGE_RESIZE_DO_VERTICALS
6148-#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6149-#include STBIR__HEADER_FILENAME
6150-
6151-#define STBIR__vertical_channels 8
6152-#define STB_IMAGE_RESIZE_DO_VERTICALS
6153-#include STBIR__HEADER_FILENAME
6154-
6155-#define STBIR__vertical_channels 8
6156-#define STB_IMAGE_RESIZE_DO_VERTICALS
6157-#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6158-#include STBIR__HEADER_FILENAME
6159-
6160-typedef void STBIR_VERTICAL_GATHERFUNC( float * output, float const * coeffs, float const ** inputs, float const * input0_end );
6161-
6162-static STBIR_VERTICAL_GATHERFUNC * stbir__vertical_gathers[ 8 ] =
6163-{
6164- stbir__vertical_gather_with_1_coeffs,stbir__vertical_gather_with_2_coeffs,stbir__vertical_gather_with_3_coeffs,stbir__vertical_gather_with_4_coeffs,stbir__vertical_gather_with_5_coeffs,stbir__vertical_gather_with_6_coeffs,stbir__vertical_gather_with_7_coeffs,stbir__vertical_gather_with_8_coeffs
6165-};
6166-
6167-static STBIR_VERTICAL_GATHERFUNC * stbir__vertical_gathers_continues[ 8 ] =
6168-{
6169- stbir__vertical_gather_with_1_coeffs_cont,stbir__vertical_gather_with_2_coeffs_cont,stbir__vertical_gather_with_3_coeffs_cont,stbir__vertical_gather_with_4_coeffs_cont,stbir__vertical_gather_with_5_coeffs_cont,stbir__vertical_gather_with_6_coeffs_cont,stbir__vertical_gather_with_7_coeffs_cont,stbir__vertical_gather_with_8_coeffs_cont
6170-};
6171-
6172-typedef void STBIR_VERTICAL_SCATTERFUNC( float ** outputs, float const * coeffs, float const * input, float const * input_end );
6173-
6174-static STBIR_VERTICAL_SCATTERFUNC * stbir__vertical_scatter_sets[ 8 ] =
6175-{
6176- stbir__vertical_scatter_with_1_coeffs,stbir__vertical_scatter_with_2_coeffs,stbir__vertical_scatter_with_3_coeffs,stbir__vertical_scatter_with_4_coeffs,stbir__vertical_scatter_with_5_coeffs,stbir__vertical_scatter_with_6_coeffs,stbir__vertical_scatter_with_7_coeffs,stbir__vertical_scatter_with_8_coeffs
6177-};
6178-
6179-static STBIR_VERTICAL_SCATTERFUNC * stbir__vertical_scatter_blends[ 8 ] =
6180-{
6181- stbir__vertical_scatter_with_1_coeffs_cont,stbir__vertical_scatter_with_2_coeffs_cont,stbir__vertical_scatter_with_3_coeffs_cont,stbir__vertical_scatter_with_4_coeffs_cont,stbir__vertical_scatter_with_5_coeffs_cont,stbir__vertical_scatter_with_6_coeffs_cont,stbir__vertical_scatter_with_7_coeffs_cont,stbir__vertical_scatter_with_8_coeffs_cont
6182-};
6183-
6184-
6185-static void stbir__encode_scanline( stbir__info const * stbir_info, void *output_buffer_data, float * encode_buffer, int row STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
6186-{
6187- int num_pixels = stbir_info->horizontal.scale_info.output_sub_size;
6188- int channels = stbir_info->channels;
6189- int width_times_channels = num_pixels * channels;
6190- void * output_buffer;
6191-
6192- // un-alpha weight if we need to
6193- if ( stbir_info->alpha_unweight )
6194- {
6195- STBIR_PROFILE_START( unalpha );
6196- stbir_info->alpha_unweight( encode_buffer, width_times_channels );
6197- STBIR_PROFILE_END( unalpha );
6198- }
6199-
6200- // write directly into output by default
6201- output_buffer = output_buffer_data;
6202-
6203- // if we have an output callback, we first convert the decode buffer in place (and then hand that to the callback)
6204- if ( stbir_info->out_pixels_cb )
6205- output_buffer = encode_buffer;
6206-
6207- STBIR_PROFILE_START( encode );
6208- // convert into the output buffer
6209- stbir_info->encode_pixels( output_buffer, width_times_channels, encode_buffer );
6210- STBIR_PROFILE_END( encode );
6211-
6212- // if we have an output callback, call it to send the data
6213- if ( stbir_info->out_pixels_cb )
6214- stbir_info->out_pixels_cb( output_buffer, num_pixels, row, stbir_info->user_data );
6215-}
6216-
6217-
6218-// Get the ring buffer pointer for an index
6219-static float* stbir__get_ring_buffer_entry(stbir__info const * stbir_info, stbir__per_split_info const * split_info, int index )
6220-{
6221- STBIR_ASSERT( index < stbir_info->ring_buffer_num_entries );
6222-
6223- #ifdef STBIR__SEPARATE_ALLOCATIONS
6224- return split_info->ring_buffers[ index ];
6225- #else
6226- return (float*) ( ( (char*) split_info->ring_buffer ) + ( index * stbir_info->ring_buffer_length_bytes ) );
6227- #endif
6228-}
6229-
6230-// Get the specified scan line from the ring buffer
6231-static float* stbir__get_ring_buffer_scanline(stbir__info const * stbir_info, stbir__per_split_info const * split_info, int get_scanline)
6232-{
6233- int ring_buffer_index = (split_info->ring_buffer_begin_index + (get_scanline - split_info->ring_buffer_first_scanline)) % stbir_info->ring_buffer_num_entries;
6234- return stbir__get_ring_buffer_entry( stbir_info, split_info, ring_buffer_index );
6235-}
6236-
6237-static void stbir__resample_horizontal_gather(stbir__info const * stbir_info, float* output_buffer, float const * input_buffer STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
6238-{
6239- float const * decode_buffer = input_buffer - ( stbir_info->scanline_extents.conservative.n0 * stbir_info->effective_channels );
6240-
6241- STBIR_PROFILE_START( horizontal );
6242- if ( ( stbir_info->horizontal.filter_enum == STBIR_FILTER_POINT_SAMPLE ) && ( stbir_info->horizontal.scale_info.scale == 1.0f ) )
6243- STBIR_MEMCPY( output_buffer, input_buffer, stbir_info->horizontal.scale_info.output_sub_size * sizeof( float ) * stbir_info->effective_channels );
6244- else
6245- stbir_info->horizontal_gather_channels( output_buffer, stbir_info->horizontal.scale_info.output_sub_size, decode_buffer, stbir_info->horizontal.contributors, stbir_info->horizontal.coefficients, stbir_info->horizontal.coefficient_width );
6246- STBIR_PROFILE_END( horizontal );
6247-}
6248-
6249-static void stbir__resample_vertical_gather(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n, int contrib_n0, int contrib_n1, float const * vertical_coefficients )
6250-{
6251- float* encode_buffer = split_info->vertical_buffer;
6252- float* decode_buffer = split_info->decode_buffer;
6253- int vertical_first = stbir_info->vertical_first;
6254- int width = (vertical_first) ? ( stbir_info->scanline_extents.conservative.n1-stbir_info->scanline_extents.conservative.n0+1 ) : stbir_info->horizontal.scale_info.output_sub_size;
6255- int width_times_channels = stbir_info->effective_channels * width;
6256-
6257- STBIR_ASSERT( stbir_info->vertical.is_gather );
6258-
6259- // loop over the contributing scanlines and scale into the buffer
6260- STBIR_PROFILE_START( vertical );
6261- {
6262- int k = 0, total = contrib_n1 - contrib_n0 + 1;
6263- STBIR_ASSERT( total > 0 );
6264- do {
6265- float const * inputs[8];
6266- int i, cnt = total; if ( cnt > 8 ) cnt = 8;
6267- for( i = 0 ; i < cnt ; i++ )
6268- inputs[ i ] = stbir__get_ring_buffer_scanline(stbir_info, split_info, k+i+contrib_n0 );
6269-
6270- // call the N scanlines at a time function (up to 8 scanlines of blending at once)
6271- ((k==0)?stbir__vertical_gathers:stbir__vertical_gathers_continues)[cnt-1]( (vertical_first) ? decode_buffer : encode_buffer, vertical_coefficients + k, inputs, inputs[0] + width_times_channels );
6272- k += cnt;
6273- total -= cnt;
6274- } while ( total );
6275- }
6276- STBIR_PROFILE_END( vertical );
6277-
6278- if ( vertical_first )
6279- {
6280- // Now resample the gathered vertical data in the horizontal axis into the encode buffer
6281- decode_buffer[ width_times_channels ] = 0.0f; // clear two over for horizontals with a remnant of 3
6282- decode_buffer[ width_times_channels+1 ] = 0.0f;
6283- stbir__resample_horizontal_gather(stbir_info, encode_buffer, decode_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6284- }
6285-
6286- stbir__encode_scanline( stbir_info, ( (char *) stbir_info->output_data ) + ((size_t)n * (size_t)stbir_info->output_stride_bytes),
6287- encode_buffer, n STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6288-}
6289-
6290-static void stbir__decode_and_resample_for_vertical_gather_loop(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n)
6291-{
6292- int ring_buffer_index;
6293- float* ring_buffer;
6294-
6295- // Decode the nth scanline from the source image into the decode buffer.
6296- stbir__decode_scanline( stbir_info, n, split_info->decode_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6297-
6298- // update new end scanline
6299- split_info->ring_buffer_last_scanline = n;
6300-
6301- // get ring buffer
6302- ring_buffer_index = (split_info->ring_buffer_begin_index + (split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline)) % stbir_info->ring_buffer_num_entries;
6303- ring_buffer = stbir__get_ring_buffer_entry(stbir_info, split_info, ring_buffer_index);
6304-
6305- // Now resample it into the ring buffer.
6306- stbir__resample_horizontal_gather( stbir_info, ring_buffer, split_info->decode_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6307-
6308- // Now it's sitting in the ring buffer ready to be used as source for the vertical sampling.
6309-}
6310-
6311-static void stbir__vertical_gather_loop( stbir__info const * stbir_info, stbir__per_split_info* split_info, int split_count )
6312-{
6313- int y, start_output_y, end_output_y;
6314- stbir__contributors* vertical_contributors = stbir_info->vertical.contributors;
6315- float const * vertical_coefficients = stbir_info->vertical.coefficients;
6316-
6317- STBIR_ASSERT( stbir_info->vertical.is_gather );
6318-
6319- start_output_y = split_info->start_output_y;
6320- end_output_y = split_info[split_count-1].end_output_y;
6321-
6322- vertical_contributors += start_output_y;
6323- vertical_coefficients += start_output_y * stbir_info->vertical.coefficient_width;
6324-
6325- // initialize the ring buffer for gathering
6326- split_info->ring_buffer_begin_index = 0;
6327- split_info->ring_buffer_first_scanline = vertical_contributors->n0;
6328- split_info->ring_buffer_last_scanline = split_info->ring_buffer_first_scanline - 1; // means "empty"
6329-
6330- for (y = start_output_y; y < end_output_y; y++)
6331- {
6332- int in_first_scanline, in_last_scanline;
6333-
6334- in_first_scanline = vertical_contributors->n0;
6335- in_last_scanline = vertical_contributors->n1;
6336-
6337- // make sure the indexing hasn't broken
6338- STBIR_ASSERT( in_first_scanline >= split_info->ring_buffer_first_scanline );
6339-
6340- // Load in new scanlines
6341- while (in_last_scanline > split_info->ring_buffer_last_scanline)
6342- {
6343- STBIR_ASSERT( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) <= stbir_info->ring_buffer_num_entries );
6344-
6345- // make sure there was room in the ring buffer when we add new scanlines
6346- if ( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) == stbir_info->ring_buffer_num_entries )
6347- {
6348- split_info->ring_buffer_first_scanline++;
6349- split_info->ring_buffer_begin_index++;
6350- }
6351-
6352- if ( stbir_info->vertical_first )
6353- {
6354- float * ring_buffer = stbir__get_ring_buffer_scanline( stbir_info, split_info, ++split_info->ring_buffer_last_scanline );
6355- // Decode the nth scanline from the source image into the decode buffer.
6356- stbir__decode_scanline( stbir_info, split_info->ring_buffer_last_scanline, ring_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6357- }
6358- else
6359- {
6360- stbir__decode_and_resample_for_vertical_gather_loop(stbir_info, split_info, split_info->ring_buffer_last_scanline + 1);
6361- }
6362- }
6363-
6364- // Now all buffers should be ready to write a row of vertical sampling, so do it.
6365- stbir__resample_vertical_gather(stbir_info, split_info, y, in_first_scanline, in_last_scanline, vertical_coefficients );
6366-
6367- ++vertical_contributors;
6368- vertical_coefficients += stbir_info->vertical.coefficient_width;
6369- }
6370-}
6371-
6372-#define STBIR__FLOAT_EMPTY_MARKER 3.0e+38F
6373-#define STBIR__FLOAT_BUFFER_IS_EMPTY(ptr) ((ptr)[0]==STBIR__FLOAT_EMPTY_MARKER)
6374-
6375-static void stbir__encode_first_scanline_from_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info)
6376-{
6377- // evict a scanline out into the output buffer
6378- float* ring_buffer_entry = stbir__get_ring_buffer_entry(stbir_info, split_info, split_info->ring_buffer_begin_index );
6379-
6380- // dump the scanline out
6381- stbir__encode_scanline( stbir_info, ( (char *)stbir_info->output_data ) + ( (size_t)split_info->ring_buffer_first_scanline * (size_t)stbir_info->output_stride_bytes ), ring_buffer_entry, split_info->ring_buffer_first_scanline STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6382-
6383- // mark it as empty
6384- ring_buffer_entry[ 0 ] = STBIR__FLOAT_EMPTY_MARKER;
6385-
6386- // advance the first scanline
6387- split_info->ring_buffer_first_scanline++;
6388- if ( ++split_info->ring_buffer_begin_index == stbir_info->ring_buffer_num_entries )
6389- split_info->ring_buffer_begin_index = 0;
6390-}
6391-
6392-static void stbir__horizontal_resample_and_encode_first_scanline_from_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info)
6393-{
6394- // evict a scanline out into the output buffer
6395-
6396- float* ring_buffer_entry = stbir__get_ring_buffer_entry(stbir_info, split_info, split_info->ring_buffer_begin_index );
6397-
6398- // Now resample it into the buffer.
6399- stbir__resample_horizontal_gather( stbir_info, split_info->vertical_buffer, ring_buffer_entry STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6400-
6401- // dump the scanline out
6402- stbir__encode_scanline( stbir_info, ( (char *)stbir_info->output_data ) + ( (size_t)split_info->ring_buffer_first_scanline * (size_t)stbir_info->output_stride_bytes ), split_info->vertical_buffer, split_info->ring_buffer_first_scanline STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6403-
6404- // mark it as empty
6405- ring_buffer_entry[ 0 ] = STBIR__FLOAT_EMPTY_MARKER;
6406-
6407- // advance the first scanline
6408- split_info->ring_buffer_first_scanline++;
6409- if ( ++split_info->ring_buffer_begin_index == stbir_info->ring_buffer_num_entries )
6410- split_info->ring_buffer_begin_index = 0;
6411-}
6412-
6413-static void stbir__resample_vertical_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n0, int n1, float const * vertical_coefficients, float const * vertical_buffer, float const * vertical_buffer_end )
6414-{
6415- STBIR_ASSERT( !stbir_info->vertical.is_gather );
6416-
6417- STBIR_PROFILE_START( vertical );
6418- {
6419- int k = 0, total = n1 - n0 + 1;
6420- STBIR_ASSERT( total > 0 );
6421- do {
6422- float * outputs[8];
6423- int i, n = total; if ( n > 8 ) n = 8;
6424- for( i = 0 ; i < n ; i++ )
6425- {
6426- outputs[ i ] = stbir__get_ring_buffer_scanline(stbir_info, split_info, k+i+n0 );
6427- if ( ( i ) && ( STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[i] ) != STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[0] ) ) ) // make sure runs are of the same type
6428- {
6429- n = i;
6430- break;
6431- }
6432- }
6433- // call the scatter to N scanlines at a time function (up to 8 scanlines of scattering at once)
6434- ((STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[0] ))?stbir__vertical_scatter_sets:stbir__vertical_scatter_blends)[n-1]( outputs, vertical_coefficients + k, vertical_buffer, vertical_buffer_end );
6435- k += n;
6436- total -= n;
6437- } while ( total );
6438- }
6439-
6440- STBIR_PROFILE_END( vertical );
6441-}
6442-
6443-typedef void stbir__handle_scanline_for_scatter_func(stbir__info const * stbir_info, stbir__per_split_info* split_info);
6444-
6445-static void stbir__vertical_scatter_loop( stbir__info const * stbir_info, stbir__per_split_info* split_info, int split_count )
6446-{
6447- int y, start_output_y, end_output_y, start_input_y, end_input_y;
6448- stbir__contributors* vertical_contributors = stbir_info->vertical.contributors;
6449- float const * vertical_coefficients = stbir_info->vertical.coefficients;
6450- stbir__handle_scanline_for_scatter_func * handle_scanline_for_scatter;
6451- void * scanline_scatter_buffer;
6452- void * scanline_scatter_buffer_end;
6453- int on_first_input_y, last_input_y;
6454- int width = (stbir_info->vertical_first) ? ( stbir_info->scanline_extents.conservative.n1-stbir_info->scanline_extents.conservative.n0+1 ) : stbir_info->horizontal.scale_info.output_sub_size;
6455- int width_times_channels = stbir_info->effective_channels * width;
6456-
6457- STBIR_ASSERT( !stbir_info->vertical.is_gather );
6458-
6459- start_output_y = split_info->start_output_y;
6460- end_output_y = split_info[split_count-1].end_output_y; // may do multiple split counts
6461-
6462- start_input_y = split_info->start_input_y;
6463- end_input_y = split_info[split_count-1].end_input_y;
6464-
6465- // adjust for starting offset start_input_y
6466- y = start_input_y + stbir_info->vertical.filter_pixel_margin;
6467- vertical_contributors += y ;
6468- vertical_coefficients += stbir_info->vertical.coefficient_width * y;
6469-
6470- if ( stbir_info->vertical_first )
6471- {
6472- handle_scanline_for_scatter = stbir__horizontal_resample_and_encode_first_scanline_from_scatter;
6473- scanline_scatter_buffer = split_info->decode_buffer;
6474- scanline_scatter_buffer_end = ( (char*) scanline_scatter_buffer ) + sizeof( float ) * stbir_info->effective_channels * (stbir_info->scanline_extents.conservative.n1-stbir_info->scanline_extents.conservative.n0+1);
6475- }
6476- else
6477- {
6478- handle_scanline_for_scatter = stbir__encode_first_scanline_from_scatter;
6479- scanline_scatter_buffer = split_info->vertical_buffer;
6480- scanline_scatter_buffer_end = ( (char*) scanline_scatter_buffer ) + sizeof( float ) * stbir_info->effective_channels * stbir_info->horizontal.scale_info.output_sub_size;
6481- }
6482-
6483- // initialize the ring buffer for scattering
6484- split_info->ring_buffer_first_scanline = start_output_y;
6485- split_info->ring_buffer_last_scanline = -1;
6486- split_info->ring_buffer_begin_index = -1;
6487-
6488- // mark all the buffers as empty to start
6489- for( y = 0 ; y < stbir_info->ring_buffer_num_entries ; y++ )
6490- {
6491- float * decode_buffer = stbir__get_ring_buffer_entry( stbir_info, split_info, y );
6492- decode_buffer[ width_times_channels ] = 0.0f; // clear two over for horizontals with a remnant of 3
6493- decode_buffer[ width_times_channels+1 ] = 0.0f;
6494- decode_buffer[0] = STBIR__FLOAT_EMPTY_MARKER; // only used on scatter
6495- }
6496-
6497- // do the loop in input space
6498- on_first_input_y = 1; last_input_y = start_input_y;
6499- for (y = start_input_y ; y < end_input_y; y++)
6500- {
6501- int out_first_scanline, out_last_scanline;
6502-
6503- out_first_scanline = vertical_contributors->n0;
6504- out_last_scanline = vertical_contributors->n1;
6505-
6506- STBIR_ASSERT(out_last_scanline - out_first_scanline + 1 <= stbir_info->ring_buffer_num_entries);
6507-
6508- if ( ( out_last_scanline >= out_first_scanline ) && ( ( ( out_first_scanline >= start_output_y ) && ( out_first_scanline < end_output_y ) ) || ( ( out_last_scanline >= start_output_y ) && ( out_last_scanline < end_output_y ) ) ) )
6509- {
6510- float const * vc = vertical_coefficients;
6511-
6512- // keep track of the range actually seen for the next resize
6513- last_input_y = y;
6514- if ( ( on_first_input_y ) && ( y > start_input_y ) )
6515- split_info->start_input_y = y;
6516- on_first_input_y = 0;
6517-
6518- // clip the region
6519- if ( out_first_scanline < start_output_y )
6520- {
6521- vc += start_output_y - out_first_scanline;
6522- out_first_scanline = start_output_y;
6523- }
6524-
6525- if ( out_last_scanline >= end_output_y )
6526- out_last_scanline = end_output_y - 1;
6527-
6528- // if very first scanline, init the index
6529- if (split_info->ring_buffer_begin_index < 0)
6530- split_info->ring_buffer_begin_index = out_first_scanline - start_output_y;
6531-
6532- STBIR_ASSERT( split_info->ring_buffer_begin_index <= out_first_scanline );
6533-
6534- // Decode the nth scanline from the source image into the decode buffer.
6535- stbir__decode_scanline( stbir_info, y, split_info->decode_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6536-
6537- // When horizontal first, we resample horizontally into the vertical buffer before we scatter it out
6538- if ( !stbir_info->vertical_first )
6539- stbir__resample_horizontal_gather( stbir_info, split_info->vertical_buffer, split_info->decode_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6540-
6541- // Now it's sitting in the buffer ready to be distributed into the ring buffers.
6542-
6543- // evict from the ringbuffer, if we need are full
6544- if ( ( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) == stbir_info->ring_buffer_num_entries ) &&
6545- ( out_last_scanline > split_info->ring_buffer_last_scanline ) )
6546- handle_scanline_for_scatter( stbir_info, split_info );
6547-
6548- // Now the horizontal buffer is ready to write to all ring buffer rows, so do it.
6549- stbir__resample_vertical_scatter(stbir_info, split_info, out_first_scanline, out_last_scanline, vc, (float*)scanline_scatter_buffer, (float*)scanline_scatter_buffer_end );
6550-
6551- // update the end of the buffer
6552- if ( out_last_scanline > split_info->ring_buffer_last_scanline )
6553- split_info->ring_buffer_last_scanline = out_last_scanline;
6554- }
6555- ++vertical_contributors;
6556- vertical_coefficients += stbir_info->vertical.coefficient_width;
6557- }
6558-
6559- // now evict the scanlines that are left over in the ring buffer
6560- while ( split_info->ring_buffer_first_scanline < end_output_y )
6561- handle_scanline_for_scatter(stbir_info, split_info);
6562-
6563- // update the end_input_y if we do multiple resizes with the same data
6564- ++last_input_y;
6565- for( y = 0 ; y < split_count; y++ )
6566- if ( split_info[y].end_input_y > last_input_y )
6567- split_info[y].end_input_y = last_input_y;
6568-}
6569-
6570-
6571-static stbir__kernel_callback * stbir__builtin_kernels[] = { 0, stbir__filter_trapezoid, stbir__filter_triangle, stbir__filter_cubic, stbir__filter_catmullrom, stbir__filter_mitchell, stbir__filter_point };
6572-static stbir__support_callback * stbir__builtin_supports[] = { 0, stbir__support_trapezoid, stbir__support_one, stbir__support_two, stbir__support_two, stbir__support_two, stbir__support_zeropoint5 };
6573-
6574-static void stbir__set_sampler(stbir__sampler * samp, stbir_filter filter, stbir__kernel_callback * kernel, stbir__support_callback * support, stbir_edge edge, stbir__scale_info * scale_info, int always_gather, void * user_data )
6575-{
6576- // set filter
6577- if (filter == 0)
6578- {
6579- filter = STBIR_DEFAULT_FILTER_DOWNSAMPLE; // default to downsample
6580- if (scale_info->scale >= ( 1.0f - stbir__small_float ) )
6581- {
6582- if ( (scale_info->scale <= ( 1.0f + stbir__small_float ) ) && ( STBIR_CEILF(scale_info->pixel_shift) == scale_info->pixel_shift ) )
6583- filter = STBIR_FILTER_POINT_SAMPLE;
6584- else
6585- filter = STBIR_DEFAULT_FILTER_UPSAMPLE;
6586- }
6587- }
6588- samp->filter_enum = filter;
6589-
6590- STBIR_ASSERT(samp->filter_enum != 0);
6591- STBIR_ASSERT((unsigned)samp->filter_enum < STBIR_FILTER_OTHER);
6592- samp->filter_kernel = stbir__builtin_kernels[ filter ];
6593- samp->filter_support = stbir__builtin_supports[ filter ];
6594-
6595- if ( kernel && support )
6596- {
6597- samp->filter_kernel = kernel;
6598- samp->filter_support = support;
6599- samp->filter_enum = STBIR_FILTER_OTHER;
6600- }
6601-
6602- samp->edge = edge;
6603- samp->filter_pixel_width = stbir__get_filter_pixel_width (samp->filter_support, scale_info->scale, user_data );
6604- // Gather is always better, but in extreme downsamples, you have to most or all of the data in memory
6605- // For horizontal, we always have all the pixels, so we always use gather here (always_gather==1).
6606- // For vertical, we use gather if scaling up (which means we will have samp->filter_pixel_width
6607- // scanlines in memory at once).
6608- samp->is_gather = 0;
6609- if ( scale_info->scale >= ( 1.0f - stbir__small_float ) )
6610- samp->is_gather = 1;
6611- else if ( ( always_gather ) || ( samp->filter_pixel_width <= STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT ) )
6612- samp->is_gather = 2;
6613-
6614- // pre calculate stuff based on the above
6615- samp->coefficient_width = stbir__get_coefficient_width(samp, samp->is_gather, user_data);
6616-
6617- // filter_pixel_width is the conservative size in pixels of input that affect an output pixel.
6618- // In rare cases (only with 2 pix to 1 pix with the default filters), it's possible that the
6619- // filter will extend before or after the scanline beyond just one extra entire copy of the
6620- // scanline (we would hit the edge twice). We don't let you do that, so we clamp the total
6621- // width to 3x the total of input pixel (once for the scanline, once for the left side
6622- // overhang, and once for the right side). We only do this for edge mode, since the other
6623- // modes can just re-edge clamp back in again.
6624- if ( edge == STBIR_EDGE_WRAP )
6625- if ( samp->filter_pixel_width > ( scale_info->input_full_size * 3 ) )
6626- samp->filter_pixel_width = scale_info->input_full_size * 3;
6627-
6628- // This is how much to expand buffers to account for filters seeking outside
6629- // the image boundaries.
6630- samp->filter_pixel_margin = samp->filter_pixel_width / 2;
6631-
6632- // filter_pixel_margin is the amount that this filter can overhang on just one side of either
6633- // end of the scanline (left or the right). Since we only allow you to overhang 1 scanline's
6634- // worth of pixels, we clamp this one side of overhang to the input scanline size. Again,
6635- // this clamping only happens in rare cases with the default filters (2 pix to 1 pix).
6636- if ( edge == STBIR_EDGE_WRAP )
6637- if ( samp->filter_pixel_margin > scale_info->input_full_size )
6638- samp->filter_pixel_margin = scale_info->input_full_size;
6639-
6640- samp->num_contributors = stbir__get_contributors(samp, samp->is_gather);
6641-
6642- samp->contributors_size = samp->num_contributors * sizeof(stbir__contributors);
6643- samp->coefficients_size = samp->num_contributors * samp->coefficient_width * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // extra sizeof(float) is padding
6644-
6645- samp->gather_prescatter_contributors = 0;
6646- samp->gather_prescatter_coefficients = 0;
6647- if ( samp->is_gather == 0 )
6648- {
6649- samp->gather_prescatter_coefficient_width = samp->filter_pixel_width;
6650- samp->gather_prescatter_num_contributors = stbir__get_contributors(samp, 2);
6651- samp->gather_prescatter_contributors_size = samp->gather_prescatter_num_contributors * sizeof(stbir__contributors);
6652- samp->gather_prescatter_coefficients_size = samp->gather_prescatter_num_contributors * samp->gather_prescatter_coefficient_width * sizeof(float);
6653- }
6654-}
6655-
6656-static void stbir__get_conservative_extents( stbir__sampler * samp, stbir__contributors * range, void * user_data )
6657-{
6658- float scale = samp->scale_info.scale;
6659- float out_shift = samp->scale_info.pixel_shift;
6660- stbir__support_callback * support = samp->filter_support;
6661- int input_full_size = samp->scale_info.input_full_size;
6662- stbir_edge edge = samp->edge;
6663- float inv_scale = samp->scale_info.inv_scale;
6664-
6665- STBIR_ASSERT( samp->is_gather != 0 );
6666-
6667- if ( samp->is_gather == 1 )
6668- {
6669- int in_first_pixel, in_last_pixel;
6670- float out_filter_radius = support(inv_scale, user_data) * scale;
6671-
6672- stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, 0.5, out_filter_radius, inv_scale, out_shift, input_full_size, edge );
6673- range->n0 = in_first_pixel;
6674- stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, ( (float)(samp->scale_info.output_sub_size-1) ) + 0.5f, out_filter_radius, inv_scale, out_shift, input_full_size, edge );
6675- range->n1 = in_last_pixel;
6676- }
6677- else if ( samp->is_gather == 2 ) // downsample gather, refine
6678- {
6679- float in_pixels_radius = support(scale, user_data) * inv_scale;
6680- int filter_pixel_margin = samp->filter_pixel_margin;
6681- int output_sub_size = samp->scale_info.output_sub_size;
6682- int input_end;
6683- int n;
6684- int in_first_pixel, in_last_pixel;
6685-
6686- // get a conservative area of the input range
6687- stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, 0, 0, inv_scale, out_shift, input_full_size, edge );
6688- range->n0 = in_first_pixel;
6689- stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, (float)output_sub_size, 0, inv_scale, out_shift, input_full_size, edge );
6690- range->n1 = in_last_pixel;
6691-
6692- // now go through the margin to the start of area to find bottom
6693- n = range->n0 + 1;
6694- input_end = -filter_pixel_margin;
6695- while( n >= input_end )
6696- {
6697- int out_first_pixel, out_last_pixel;
6698- stbir__calculate_out_pixel_range( &out_first_pixel, &out_last_pixel, ((float)n)+0.5f, in_pixels_radius, scale, out_shift, output_sub_size );
6699- if ( out_first_pixel > out_last_pixel )
6700- break;
6701-
6702- if ( ( out_first_pixel < output_sub_size ) || ( out_last_pixel >= 0 ) )
6703- range->n0 = n;
6704- --n;
6705- }
6706-
6707- // now go through the end of the area through the margin to find top
6708- n = range->n1 - 1;
6709- input_end = n + 1 + filter_pixel_margin;
6710- while( n <= input_end )
6711- {
6712- int out_first_pixel, out_last_pixel;
6713- stbir__calculate_out_pixel_range( &out_first_pixel, &out_last_pixel, ((float)n)+0.5f, in_pixels_radius, scale, out_shift, output_sub_size );
6714- if ( out_first_pixel > out_last_pixel )
6715- break;
6716- if ( ( out_first_pixel < output_sub_size ) || ( out_last_pixel >= 0 ) )
6717- range->n1 = n;
6718- ++n;
6719- }
6720- }
6721-
6722- if ( samp->edge == STBIR_EDGE_WRAP )
6723- {
6724- // if we are wrapping, and we are very close to the image size (so the edges might merge), just use the scanline up to the edge
6725- if ( ( range->n0 > 0 ) && ( range->n1 >= input_full_size ) )
6726- {
6727- int marg = range->n1 - input_full_size + 1;
6728- if ( ( marg + STBIR__MERGE_RUNS_PIXEL_THRESHOLD ) >= range->n0 )
6729- range->n0 = 0;
6730- }
6731- if ( ( range->n0 < 0 ) && ( range->n1 < (input_full_size-1) ) )
6732- {
6733- int marg = -range->n0;
6734- if ( ( input_full_size - marg - STBIR__MERGE_RUNS_PIXEL_THRESHOLD - 1 ) <= range->n1 )
6735- range->n1 = input_full_size - 1;
6736- }
6737- }
6738- else
6739- {
6740- // for non-edge-wrap modes, we never read over the edge, so clamp
6741- if ( range->n0 < 0 )
6742- range->n0 = 0;
6743- if ( range->n1 >= input_full_size )
6744- range->n1 = input_full_size - 1;
6745- }
6746-}
6747-
6748-static void stbir__get_split_info( stbir__per_split_info* split_info, int splits, int output_height, int vertical_pixel_margin, int input_full_height, int is_gather, stbir__contributors * contribs )
6749-{
6750- int i, cur;
6751- int left = output_height;
6752-
6753- cur = 0;
6754- for( i = 0 ; i < splits ; i++ )
6755- {
6756- int each;
6757-
6758- split_info[i].start_output_y = cur;
6759- each = left / ( splits - i );
6760- split_info[i].end_output_y = cur + each;
6761-
6762- // ok, when we are gathering, we need to make sure we are starting on a y offset that doesn't have
6763- // a "special" set of coefficients. Basically, with exactly the right filter at exactly the right
6764- // resize at exactly the right phase, some of the coefficents can be zero. When they are zero, we
6765- // don't process them at all. But this leads to a tricky thing with the thread splits, where we
6766- // might have a set of two coeffs like this for example: (4,4) and (3,6). The 4,4 means there was
6767- // just one single coeff because things worked out perfectly (normally, they all have 4 coeffs
6768- // like the range 3,6. The problem is that if we start right on the (4,4) on a brand new thread,
6769- // then when we get to (3,6), we don't have the "3" sample in memory (because we didn't load
6770- // it on the initial (4,4) range because it didn't have a 3 (we only add new samples that are
6771- // larger than our existing samples - it's just how the eviction works). So, our solution here
6772- // is pretty simple, if we start right on a range that has samples that start earlier, then we
6773- // simply bump up our previous thread split range to include it, and then start this threads
6774- // range with the smaller sample. It just moves one scanline from one thread split to another,
6775- // so that we end with the unusual one, instead of start with it. To do this, we check 2-4
6776- // sample at each thread split start and then occassionally move them.
6777-
6778- if ( ( is_gather ) && ( i ) )
6779- {
6780- stbir__contributors * small_contribs;
6781- int j, smallest, stop, start_n0;
6782- stbir__contributors * split_contribs = contribs + cur;
6783-
6784- // scan for a max of 3x the filter width or until the next thread split
6785- stop = vertical_pixel_margin * 3;
6786- if ( each < stop )
6787- stop = each;
6788-
6789- // loops a few times before early out
6790- smallest = 0;
6791- small_contribs = split_contribs;
6792- start_n0 = small_contribs->n0;
6793- for( j = 1 ; j <= stop ; j++ )
6794- {
6795- ++split_contribs;
6796- if ( split_contribs->n0 > start_n0 )
6797- break;
6798- if ( split_contribs->n0 < small_contribs->n0 )
6799- {
6800- small_contribs = split_contribs;
6801- smallest = j;
6802- }
6803- }
6804-
6805- split_info[i-1].end_output_y += smallest;
6806- split_info[i].start_output_y += smallest;
6807- }
6808-
6809- cur += each;
6810- left -= each;
6811-
6812- // scatter range (updated to minimum as you run it)
6813- split_info[i].start_input_y = -vertical_pixel_margin;
6814- split_info[i].end_input_y = input_full_height + vertical_pixel_margin;
6815- }
6816-}
6817-
6818-static void stbir__free_internal_mem( stbir__info *info )
6819-{
6820- #define STBIR__FREE_AND_CLEAR( ptr ) { if ( ptr ) { void * p = (ptr); (ptr) = 0; STBIR_FREE( p, info->user_data); } }
6821-
6822- if ( info )
6823- {
6824- #ifndef STBIR__SEPARATE_ALLOCATIONS
6825- STBIR__FREE_AND_CLEAR( info->alloced_mem );
6826- #else
6827- int i,j;
6828-
6829- if ( ( info->vertical.gather_prescatter_contributors ) && ( (void*)info->vertical.gather_prescatter_contributors != (void*)info->split_info[0].decode_buffer ) )
6830- {
6831- STBIR__FREE_AND_CLEAR( info->vertical.gather_prescatter_coefficients );
6832- STBIR__FREE_AND_CLEAR( info->vertical.gather_prescatter_contributors );
6833- }
6834- for( i = 0 ; i < info->splits ; i++ )
6835- {
6836- for( j = 0 ; j < info->alloc_ring_buffer_num_entries ; j++ )
6837- {
6838- #ifdef STBIR_SIMD8
6839- if ( info->effective_channels == 3 )
6840- --info->split_info[i].ring_buffers[j]; // avx in 3 channel mode needs one float at the start of the buffer
6841- #endif
6842- STBIR__FREE_AND_CLEAR( info->split_info[i].ring_buffers[j] );
6843- }
6844-
6845- #ifdef STBIR_SIMD8
6846- if ( info->effective_channels == 3 )
6847- --info->split_info[i].decode_buffer; // avx in 3 channel mode needs one float at the start of the buffer
6848- #endif
6849- STBIR__FREE_AND_CLEAR( info->split_info[i].decode_buffer );
6850- STBIR__FREE_AND_CLEAR( info->split_info[i].ring_buffers );
6851- STBIR__FREE_AND_CLEAR( info->split_info[i].vertical_buffer );
6852- }
6853- STBIR__FREE_AND_CLEAR( info->split_info );
6854- if ( info->vertical.coefficients != info->horizontal.coefficients )
6855- {
6856- STBIR__FREE_AND_CLEAR( info->vertical.coefficients );
6857- STBIR__FREE_AND_CLEAR( info->vertical.contributors );
6858- }
6859- STBIR__FREE_AND_CLEAR( info->horizontal.coefficients );
6860- STBIR__FREE_AND_CLEAR( info->horizontal.contributors );
6861- STBIR__FREE_AND_CLEAR( info->alloced_mem );
6862- STBIR_FREE( info, info->user_data );
6863- #endif
6864- }
6865-
6866- #undef STBIR__FREE_AND_CLEAR
6867-}
6868-
6869-static int stbir__get_max_split( int splits, int height )
6870-{
6871- int i;
6872- int max = 0;
6873-
6874- for( i = 0 ; i < splits ; i++ )
6875- {
6876- int each = height / ( splits - i );
6877- if ( each > max )
6878- max = each;
6879- height -= each;
6880- }
6881- return max;
6882-}
6883-
6884-static stbir__horizontal_gather_channels_func ** stbir__horizontal_gather_n_coeffs_funcs[8] =
6885-{
6886- 0, stbir__horizontal_gather_1_channels_with_n_coeffs_funcs, stbir__horizontal_gather_2_channels_with_n_coeffs_funcs, stbir__horizontal_gather_3_channels_with_n_coeffs_funcs, stbir__horizontal_gather_4_channels_with_n_coeffs_funcs, 0,0, stbir__horizontal_gather_7_channels_with_n_coeffs_funcs
6887-};
6888-
6889-static stbir__horizontal_gather_channels_func ** stbir__horizontal_gather_channels_funcs[8] =
6890-{
6891- 0, stbir__horizontal_gather_1_channels_funcs, stbir__horizontal_gather_2_channels_funcs, stbir__horizontal_gather_3_channels_funcs, stbir__horizontal_gather_4_channels_funcs, 0,0, stbir__horizontal_gather_7_channels_funcs
6892-};
6893-
6894-// there are six resize classifications: 0 == vertical scatter, 1 == vertical gather < 1x scale, 2 == vertical gather 1x-2x scale, 4 == vertical gather < 3x scale, 4 == vertical gather > 3x scale, 5 == <=4 pixel height, 6 == <=4 pixel wide column
6895-#define STBIR_RESIZE_CLASSIFICATIONS 8
6896-
6897-static float stbir__compute_weights[5][STBIR_RESIZE_CLASSIFICATIONS][4]= // 5 = 0=1chan, 1=2chan, 2=3chan, 3=4chan, 4=7chan
6898-{
6899- {
6900- { 1.00000f, 1.00000f, 0.31250f, 1.00000f },
6901- { 0.56250f, 0.59375f, 0.00000f, 0.96875f },
6902- { 1.00000f, 0.06250f, 0.00000f, 1.00000f },
6903- { 0.00000f, 0.09375f, 1.00000f, 1.00000f },
6904- { 1.00000f, 1.00000f, 0.31250f, 1.00000f },
6905- { 0.03125f, 0.12500f, 1.00000f, 1.00000f },
6906- { 1.00000f, 1.00000f, 0.06250f, 1.00000f },
6907- { 0.00000f, 1.00000f, 0.00000f, 0.03125f },
6908- }, {
6909- { 0.00000f, 0.84375f, 0.00000f, 0.03125f },
6910- { 0.09375f, 0.93750f, 0.00000f, 0.78125f },
6911- { 0.87500f, 0.21875f, 0.00000f, 0.96875f },
6912- { 0.09375f, 0.09375f, 1.00000f, 1.00000f },
6913- { 0.00000f, 0.84375f, 0.00000f, 0.03125f },
6914- { 0.03125f, 0.12500f, 1.00000f, 1.00000f },
6915- { 1.00000f, 1.00000f, 0.06250f, 1.00000f },
6916- { 0.00000f, 1.00000f, 0.00000f, 0.53125f },
6917- }, {
6918- { 0.00000f, 0.53125f, 0.00000f, 0.03125f },
6919- { 0.06250f, 0.96875f, 0.00000f, 0.53125f },
6920- { 0.87500f, 0.18750f, 0.00000f, 0.93750f },
6921- { 0.00000f, 0.09375f, 1.00000f, 1.00000f },
6922- { 0.00000f, 0.53125f, 0.00000f, 0.03125f },
6923- { 0.03125f, 0.12500f, 1.00000f, 1.00000f },
6924- { 1.00000f, 1.00000f, 0.06250f, 1.00000f },
6925- { 0.00000f, 1.00000f, 0.00000f, 0.56250f },
6926- }, {
6927- { 0.00000f, 0.50000f, 0.00000f, 0.71875f },
6928- { 0.06250f, 0.84375f, 0.00000f, 0.87500f },
6929- { 1.00000f, 0.50000f, 0.50000f, 0.96875f },
6930- { 1.00000f, 0.09375f, 0.31250f, 0.50000f },
6931- { 0.00000f, 0.50000f, 0.00000f, 0.71875f },
6932- { 1.00000f, 0.03125f, 0.03125f, 0.53125f },
6933- { 1.00000f, 1.00000f, 0.06250f, 1.00000f },
6934- { 0.00000f, 1.00000f, 0.03125f, 0.18750f },
6935- }, {
6936- { 0.00000f, 0.59375f, 0.00000f, 0.96875f },
6937- { 0.06250f, 0.81250f, 0.06250f, 0.59375f },
6938- { 0.75000f, 0.43750f, 0.12500f, 0.96875f },
6939- { 0.87500f, 0.06250f, 0.18750f, 0.43750f },
6940- { 0.00000f, 0.59375f, 0.00000f, 0.96875f },
6941- { 0.15625f, 0.12500f, 1.00000f, 1.00000f },
6942- { 1.00000f, 1.00000f, 0.06250f, 1.00000f },
6943- { 0.00000f, 1.00000f, 0.03125f, 0.34375f },
6944- }
6945-};
6946-
6947-// structure that allow us to query and override info for training the costs
6948-typedef struct STBIR__V_FIRST_INFO
6949-{
6950- double v_cost, h_cost;
6951- int control_v_first; // 0 = no control, 1 = force hori, 2 = force vert
6952- int v_first;
6953- int v_resize_classification;
6954- int is_gather;
6955-} STBIR__V_FIRST_INFO;
6956-
6957-#ifdef STBIR__V_FIRST_INFO_BUFFER
6958-static STBIR__V_FIRST_INFO STBIR__V_FIRST_INFO_BUFFER = {0};
6959-#define STBIR__V_FIRST_INFO_POINTER &STBIR__V_FIRST_INFO_BUFFER
6960-#else
6961-#define STBIR__V_FIRST_INFO_POINTER 0
6962-#endif
6963-
6964-// Figure out whether to scale along the horizontal or vertical first.
6965-// This only *super* important when you are scaling by a massively
6966-// different amount in the vertical vs the horizontal (for example, if
6967-// you are scaling by 2x in the width, and 0.5x in the height, then you
6968-// want to do the vertical scale first, because it's around 3x faster
6969-// in that order.
6970-//
6971-// In more normal circumstances, this makes a 20-40% differences, so
6972-// it's good to get right, but not critical. The normal way that you
6973-// decide which direction goes first is just figuring out which
6974-// direction does more multiplies. But with modern CPUs with their
6975-// fancy caches and SIMD and high IPC abilities, so there's just a lot
6976-// more that goes into it.
6977-//
6978-// My handwavy sort of solution is to have an app that does a whole
6979-// bunch of timing for both vertical and horizontal first modes,
6980-// and then another app that can read lots of these timing files
6981-// and try to search for the best weights to use. Dotimings.c
6982-// is the app that does a bunch of timings, and vf_train.c is the
6983-// app that solves for the best weights (and shows how well it
6984-// does currently).
6985-
6986-static int stbir__should_do_vertical_first( float weights_table[STBIR_RESIZE_CLASSIFICATIONS][4], int horizontal_filter_pixel_width, float horizontal_scale, int horizontal_output_size, int vertical_filter_pixel_width, float vertical_scale, int vertical_output_size, int is_gather, STBIR__V_FIRST_INFO * info )
6987-{
6988- double v_cost, h_cost;
6989- float * weights;
6990- int vertical_first;
6991- int v_classification;
6992-
6993- // categorize the resize into buckets
6994- if ( ( vertical_output_size <= 4 ) || ( horizontal_output_size <= 4 ) )
6995- v_classification = ( vertical_output_size < horizontal_output_size ) ? 6 : 7;
6996- else if ( ( !is_gather ) && ( ( vertical_output_size <= 16 ) || ( horizontal_output_size <= 16 ) ) )
6997- v_classification = 4;
6998- else if ( vertical_scale <= 1.0f )
6999- v_classification = ( is_gather ) ? 1 : 0;
7000- else if ( vertical_scale <= 2.0f)
7001- v_classification = 2;
7002- else if ( vertical_scale <= 3.0f)
7003- v_classification = 3;
7004- else
7005- v_classification = 5; // everything bigger than 3x
7006-
7007- // use the right weights
7008- weights = weights_table[ v_classification ];
7009-
7010- // this is the costs when you don't take into account modern CPUs with high ipc and simd and caches - wish we had a better estimate
7011- h_cost = (float)horizontal_filter_pixel_width * weights[0] + horizontal_scale * (float)vertical_filter_pixel_width * weights[1];
7012- v_cost = (float)vertical_filter_pixel_width * weights[2] + vertical_scale * (float)horizontal_filter_pixel_width * weights[3];
7013-
7014- // use computation estimate to decide vertical first or not
7015- vertical_first = ( v_cost <= h_cost ) ? 1 : 0;
7016-
7017- // save these, if requested
7018- if ( info )
7019- {
7020- info->h_cost = h_cost;
7021- info->v_cost = v_cost;
7022- info->v_resize_classification = v_classification;
7023- info->v_first = vertical_first;
7024- info->is_gather = is_gather;
7025- }
7026-
7027- // and this allows us to override everything for testing (see dotiming.c)
7028- if ( ( info ) && ( info->control_v_first ) )
7029- vertical_first = ( info->control_v_first == 2 ) ? 1 : 0;
7030-
7031- return vertical_first;
7032-}
7033-
7034-// layout lookups - must match stbir_internal_pixel_layout
7035-static unsigned char stbir__pixel_channels[] = {
7036- 1,2,3,3,4, // 1ch, 2ch, rgb, bgr, 4ch
7037- 4,4,4,4,2,2, // RGBA,BGRA,ARGB,ABGR,RA,AR
7038- 4,4,4,4,2,2, // RGBA_PM,BGRA_PM,ARGB_PM,ABGR_PM,RA_PM,AR_PM
7039-};
7040-
7041-// the internal pixel layout enums are in a different order, so we can easily do range comparisons of types
7042-// the public pixel layout is ordered in a way that if you cast num_channels (1-4) to the enum, you get something sensible
7043-static stbir_internal_pixel_layout stbir__pixel_layout_convert_public_to_internal[] = {
7044- STBIRI_BGR, STBIRI_1CHANNEL, STBIRI_2CHANNEL, STBIRI_RGB, STBIRI_RGBA,
7045- STBIRI_4CHANNEL, STBIRI_BGRA, STBIRI_ARGB, STBIRI_ABGR, STBIRI_RA, STBIRI_AR,
7046- STBIRI_RGBA_PM, STBIRI_BGRA_PM, STBIRI_ARGB_PM, STBIRI_ABGR_PM, STBIRI_RA_PM, STBIRI_AR_PM,
7047-};
7048-
7049-static stbir__info * stbir__alloc_internal_mem_and_build_samplers( stbir__sampler * horizontal, stbir__sampler * vertical, stbir__contributors * conservative, stbir_pixel_layout input_pixel_layout_public, stbir_pixel_layout output_pixel_layout_public, int splits, int new_x, int new_y, int fast_alpha, void * user_data STBIR_ONLY_PROFILE_BUILD_GET_INFO )
7050-{
7051- static char stbir_channel_count_index[8]={ 9,0,1,2, 3,9,9,4 };
7052-
7053- stbir__info * info = 0;
7054- void * alloced = 0;
7055- size_t alloced_total = 0;
7056- int vertical_first;
7057- size_t decode_buffer_size, ring_buffer_length_bytes, ring_buffer_size, vertical_buffer_size;
7058- int alloc_ring_buffer_num_entries;
7059-
7060- int alpha_weighting_type = 0; // 0=none, 1=simple, 2=fancy
7061- int conservative_split_output_size = stbir__get_max_split( splits, vertical->scale_info.output_sub_size );
7062- stbir_internal_pixel_layout input_pixel_layout = stbir__pixel_layout_convert_public_to_internal[ input_pixel_layout_public ];
7063- stbir_internal_pixel_layout output_pixel_layout = stbir__pixel_layout_convert_public_to_internal[ output_pixel_layout_public ];
7064- int channels = stbir__pixel_channels[ input_pixel_layout ];
7065- int effective_channels = channels;
7066-
7067- // first figure out what type of alpha weighting to use (if any)
7068- if ( ( horizontal->filter_enum != STBIR_FILTER_POINT_SAMPLE ) || ( vertical->filter_enum != STBIR_FILTER_POINT_SAMPLE ) ) // no alpha weighting on point sampling
7069- {
7070- if ( ( input_pixel_layout >= STBIRI_RGBA ) && ( input_pixel_layout <= STBIRI_AR ) && ( output_pixel_layout >= STBIRI_RGBA ) && ( output_pixel_layout <= STBIRI_AR ) )
7071- {
7072- if ( fast_alpha )
7073- {
7074- alpha_weighting_type = 4;
7075- }
7076- else
7077- {
7078- static int fancy_alpha_effective_cnts[6] = { 7, 7, 7, 7, 3, 3 };
7079- alpha_weighting_type = 2;
7080- effective_channels = fancy_alpha_effective_cnts[ input_pixel_layout - STBIRI_RGBA ];
7081- }
7082- }
7083- else if ( ( input_pixel_layout >= STBIRI_RGBA_PM ) && ( input_pixel_layout <= STBIRI_AR_PM ) && ( output_pixel_layout >= STBIRI_RGBA ) && ( output_pixel_layout <= STBIRI_AR ) )
7084- {
7085- // input premult, output non-premult
7086- alpha_weighting_type = 3;
7087- }
7088- else if ( ( input_pixel_layout >= STBIRI_RGBA ) && ( input_pixel_layout <= STBIRI_AR ) && ( output_pixel_layout >= STBIRI_RGBA_PM ) && ( output_pixel_layout <= STBIRI_AR_PM ) )
7089- {
7090- // input non-premult, output premult
7091- alpha_weighting_type = 1;
7092- }
7093- }
7094-
7095- // channel in and out count must match currently
7096- if ( channels != stbir__pixel_channels[ output_pixel_layout ] )
7097- return 0;
7098-
7099- // get vertical first
7100- vertical_first = stbir__should_do_vertical_first( stbir__compute_weights[ (int)stbir_channel_count_index[ effective_channels ] ], horizontal->filter_pixel_width, horizontal->scale_info.scale, horizontal->scale_info.output_sub_size, vertical->filter_pixel_width, vertical->scale_info.scale, vertical->scale_info.output_sub_size, vertical->is_gather, STBIR__V_FIRST_INFO_POINTER );
7101-
7102- // sometimes read one float off in some of the unrolled loops (with a weight of zero coeff, so it doesn't have an effect)
7103- // we use a few extra floats instead of just 1, so that input callback buffer can overlap with the decode buffer without
7104- // the conversion routines overwriting the callback input data.
7105- decode_buffer_size = ( conservative->n1 - conservative->n0 + 1 ) * effective_channels * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // extra floats for input callback stagger
7106-
7107-#if defined( STBIR__SEPARATE_ALLOCATIONS ) && defined(STBIR_SIMD8)
7108- if ( effective_channels == 3 )
7109- decode_buffer_size += sizeof(float); // avx in 3 channel mode needs one float at the start of the buffer (only with separate allocations)
7110-#endif
7111-
7112- ring_buffer_length_bytes = (size_t)horizontal->scale_info.output_sub_size * (size_t)effective_channels * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // extra floats for padding
7113-
7114- // if we do vertical first, the ring buffer holds a whole decoded line
7115- if ( vertical_first )
7116- ring_buffer_length_bytes = ( decode_buffer_size + 15 ) & ~15;
7117-
7118- if ( ( ring_buffer_length_bytes & 4095 ) == 0 ) ring_buffer_length_bytes += 64*3; // avoid 4k alias
7119-
7120- // One extra entry because floating point precision problems sometimes cause an extra to be necessary.
7121- alloc_ring_buffer_num_entries = vertical->filter_pixel_width + 1;
7122-
7123- // we never need more ring buffer entries than the scanlines we're outputting when in scatter mode
7124- if ( ( !vertical->is_gather ) && ( alloc_ring_buffer_num_entries > conservative_split_output_size ) )
7125- alloc_ring_buffer_num_entries = conservative_split_output_size;
7126-
7127- ring_buffer_size = (size_t)alloc_ring_buffer_num_entries * (size_t)ring_buffer_length_bytes;
7128-
7129- // The vertical buffer is used differently, depending on whether we are scattering
7130- // the vertical scanlines, or gathering them.
7131- // If scattering, it's used at the temp buffer to accumulate each output.
7132- // If gathering, it's just the output buffer.
7133- vertical_buffer_size = (size_t)horizontal->scale_info.output_sub_size * (size_t)effective_channels * sizeof(float) + sizeof(float); // extra float for padding
7134-
7135- // we make two passes through this loop, 1st to add everything up, 2nd to allocate and init
7136- for(;;)
7137- {
7138- int i;
7139- void * advance_mem = alloced;
7140- int copy_horizontal = 0;
7141- stbir__sampler * possibly_use_horizontal_for_pivot = 0;
7142-
7143-#ifdef STBIR__SEPARATE_ALLOCATIONS
7144- #define STBIR__NEXT_PTR( ptr, size, ntype ) if ( alloced ) { void * p = STBIR_MALLOC( size, user_data); if ( p == 0 ) { stbir__free_internal_mem( info ); return 0; } (ptr) = (ntype*)p; }
7145-#else
7146- #define STBIR__NEXT_PTR( ptr, size, ntype ) advance_mem = (void*) ( ( ((size_t)advance_mem) + 15 ) & ~15 ); if ( alloced ) ptr = (ntype*)advance_mem; advance_mem = (char*)(((size_t)advance_mem) + (size));
7147-#endif
7148-
7149- STBIR__NEXT_PTR( info, sizeof( stbir__info ), stbir__info );
7150-
7151- STBIR__NEXT_PTR( info->split_info, sizeof( stbir__per_split_info ) * splits, stbir__per_split_info );
7152-
7153- if ( info )
7154- {
7155- static stbir__alpha_weight_func * fancy_alpha_weights[6] = { stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_2ch, stbir__fancy_alpha_weight_2ch };
7156- static stbir__alpha_unweight_func * fancy_alpha_unweights[6] = { stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_2ch, stbir__fancy_alpha_unweight_2ch };
7157- static stbir__alpha_weight_func * simple_alpha_weights[6] = { stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_2ch, stbir__simple_alpha_weight_2ch };
7158- static stbir__alpha_unweight_func * simple_alpha_unweights[6] = { stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_2ch, stbir__simple_alpha_unweight_2ch };
7159-
7160- // initialize info fields
7161- info->alloced_mem = alloced;
7162- info->alloced_total = alloced_total;
7163-
7164- info->channels = channels;
7165- info->effective_channels = effective_channels;
7166-
7167- info->offset_x = new_x;
7168- info->offset_y = new_y;
7169- info->alloc_ring_buffer_num_entries = (int)alloc_ring_buffer_num_entries;
7170- info->ring_buffer_num_entries = 0;
7171- info->ring_buffer_length_bytes = (int)ring_buffer_length_bytes;
7172- info->splits = splits;
7173- info->vertical_first = vertical_first;
7174-
7175- info->input_pixel_layout_internal = input_pixel_layout;
7176- info->output_pixel_layout_internal = output_pixel_layout;
7177-
7178- // setup alpha weight functions
7179- info->alpha_weight = 0;
7180- info->alpha_unweight = 0;
7181-
7182- // handle alpha weighting functions and overrides
7183- if ( alpha_weighting_type == 2 )
7184- {
7185- // high quality alpha multiplying on the way in, dividing on the way out
7186- info->alpha_weight = fancy_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
7187- info->alpha_unweight = fancy_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
7188- }
7189- else if ( alpha_weighting_type == 4 )
7190- {
7191- // fast alpha multiplying on the way in, dividing on the way out
7192- info->alpha_weight = simple_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
7193- info->alpha_unweight = simple_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
7194- }
7195- else if ( alpha_weighting_type == 1 )
7196- {
7197- // fast alpha on the way in, leave in premultiplied form on way out
7198- info->alpha_weight = simple_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
7199- }
7200- else if ( alpha_weighting_type == 3 )
7201- {
7202- // incoming is premultiplied, fast alpha dividing on the way out - non-premultiplied output
7203- info->alpha_unweight = simple_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
7204- }
7205-
7206- // handle 3-chan color flipping, using the alpha weight path
7207- if ( ( ( input_pixel_layout == STBIRI_RGB ) && ( output_pixel_layout == STBIRI_BGR ) ) ||
7208- ( ( input_pixel_layout == STBIRI_BGR ) && ( output_pixel_layout == STBIRI_RGB ) ) )
7209- {
7210- // do the flipping on the smaller of the two ends
7211- if ( horizontal->scale_info.scale < 1.0f )
7212- info->alpha_unweight = stbir__simple_flip_3ch;
7213- else
7214- info->alpha_weight = stbir__simple_flip_3ch;
7215- }
7216-
7217- }
7218-
7219- // get all the per-split buffers
7220- for( i = 0 ; i < splits ; i++ )
7221- {
7222- STBIR__NEXT_PTR( info->split_info[i].decode_buffer, decode_buffer_size, float );
7223-
7224-#ifdef STBIR__SEPARATE_ALLOCATIONS
7225-
7226- #ifdef STBIR_SIMD8
7227- if ( ( info ) && ( effective_channels == 3 ) )
7228- ++info->split_info[i].decode_buffer; // avx in 3 channel mode needs one float at the start of the buffer
7229- #endif
7230-
7231- STBIR__NEXT_PTR( info->split_info[i].ring_buffers, alloc_ring_buffer_num_entries * sizeof(float*), float* );
7232- {
7233- int j;
7234- for( j = 0 ; j < alloc_ring_buffer_num_entries ; j++ )
7235- {
7236- STBIR__NEXT_PTR( info->split_info[i].ring_buffers[j], ring_buffer_length_bytes, float );
7237- #ifdef STBIR_SIMD8
7238- if ( ( info ) && ( effective_channels == 3 ) )
7239- ++info->split_info[i].ring_buffers[j]; // avx in 3 channel mode needs one float at the start of the buffer
7240- #endif
7241- }
7242- }
7243-#else
7244- STBIR__NEXT_PTR( info->split_info[i].ring_buffer, ring_buffer_size, float );
7245-#endif
7246- STBIR__NEXT_PTR( info->split_info[i].vertical_buffer, vertical_buffer_size, float );
7247- }
7248-
7249- // alloc memory for to-be-pivoted coeffs (if necessary)
7250- if ( vertical->is_gather == 0 )
7251- {
7252- size_t both;
7253- size_t temp_mem_amt;
7254-
7255- // when in vertical scatter mode, we first build the coefficients in gather mode, and then pivot after,
7256- // that means we need two buffers, so we try to use the decode buffer and ring buffer for this. if that
7257- // is too small, we just allocate extra memory to use as this temp.
7258-
7259- both = (size_t)vertical->gather_prescatter_contributors_size + (size_t)vertical->gather_prescatter_coefficients_size;
7260-
7261-#ifdef STBIR__SEPARATE_ALLOCATIONS
7262- temp_mem_amt = decode_buffer_size;
7263-
7264- #ifdef STBIR_SIMD8
7265- if ( effective_channels == 3 )
7266- --temp_mem_amt; // avx in 3 channel mode needs one float at the start of the buffer
7267- #endif
7268-#else
7269- temp_mem_amt = (size_t)( decode_buffer_size + ring_buffer_size + vertical_buffer_size ) * (size_t)splits;
7270-#endif
7271- if ( temp_mem_amt >= both )
7272- {
7273- if ( info )
7274- {
7275- vertical->gather_prescatter_contributors = (stbir__contributors*)info->split_info[0].decode_buffer;
7276- vertical->gather_prescatter_coefficients = (float*) ( ( (char*)info->split_info[0].decode_buffer ) + vertical->gather_prescatter_contributors_size );
7277- }
7278- }
7279- else
7280- {
7281- // ring+decode memory is too small, so allocate temp memory
7282- STBIR__NEXT_PTR( vertical->gather_prescatter_contributors, vertical->gather_prescatter_contributors_size, stbir__contributors );
7283- STBIR__NEXT_PTR( vertical->gather_prescatter_coefficients, vertical->gather_prescatter_coefficients_size, float );
7284- }
7285- }
7286-
7287- STBIR__NEXT_PTR( horizontal->contributors, horizontal->contributors_size, stbir__contributors );
7288- STBIR__NEXT_PTR( horizontal->coefficients, horizontal->coefficients_size, float );
7289-
7290- // are the two filters identical?? (happens a lot with mipmap generation)
7291- if ( ( horizontal->filter_kernel == vertical->filter_kernel ) && ( horizontal->filter_support == vertical->filter_support ) && ( horizontal->edge == vertical->edge ) && ( horizontal->scale_info.output_sub_size == vertical->scale_info.output_sub_size ) )
7292- {
7293- float diff_scale = horizontal->scale_info.scale - vertical->scale_info.scale;
7294- float diff_shift = horizontal->scale_info.pixel_shift - vertical->scale_info.pixel_shift;
7295- if ( diff_scale < 0.0f ) diff_scale = -diff_scale;
7296- if ( diff_shift < 0.0f ) diff_shift = -diff_shift;
7297- if ( ( diff_scale <= stbir__small_float ) && ( diff_shift <= stbir__small_float ) )
7298- {
7299- if ( horizontal->is_gather == vertical->is_gather )
7300- {
7301- copy_horizontal = 1;
7302- goto no_vert_alloc;
7303- }
7304- // everything matches, but vertical is scatter, horizontal is gather, use horizontal coeffs for vertical pivot coeffs
7305- possibly_use_horizontal_for_pivot = horizontal;
7306- }
7307- }
7308-
7309- STBIR__NEXT_PTR( vertical->contributors, vertical->contributors_size, stbir__contributors );
7310- STBIR__NEXT_PTR( vertical->coefficients, vertical->coefficients_size, float );
7311-
7312- no_vert_alloc:
7313-
7314- if ( info )
7315- {
7316- STBIR_PROFILE_BUILD_START( horizontal );
7317-
7318- stbir__calculate_filters( horizontal, 0, user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
7319-
7320- // setup the horizontal gather functions
7321- // start with defaulting to the n_coeffs functions (specialized on channels and remnant leftover)
7322- info->horizontal_gather_channels = stbir__horizontal_gather_n_coeffs_funcs[ effective_channels ][ horizontal->extent_info.widest & 3 ];
7323- // but if the number of coeffs <= 12, use another set of special cases. <=12 coeffs is any enlarging resize, or shrinking resize down to about 1/3 size
7324- if ( horizontal->extent_info.widest <= 12 )
7325- info->horizontal_gather_channels = stbir__horizontal_gather_channels_funcs[ effective_channels ][ horizontal->extent_info.widest - 1 ];
7326-
7327- info->scanline_extents.conservative.n0 = conservative->n0;
7328- info->scanline_extents.conservative.n1 = conservative->n1;
7329-
7330- // get exact extents
7331- stbir__get_extents( horizontal, &info->scanline_extents );
7332-
7333- // pack the horizontal coeffs
7334- horizontal->coefficient_width = stbir__pack_coefficients(horizontal->num_contributors, horizontal->contributors, horizontal->coefficients, horizontal->coefficient_width, horizontal->extent_info.widest, info->scanline_extents.conservative.n0, info->scanline_extents.conservative.n1 );
7335-
7336- STBIR_MEMCPY( &info->horizontal, horizontal, sizeof( stbir__sampler ) );
7337-
7338- STBIR_PROFILE_BUILD_END( horizontal );
7339-
7340- if ( copy_horizontal )
7341- {
7342- STBIR_MEMCPY( &info->vertical, horizontal, sizeof( stbir__sampler ) );
7343- }
7344- else
7345- {
7346- STBIR_PROFILE_BUILD_START( vertical );
7347-
7348- stbir__calculate_filters( vertical, possibly_use_horizontal_for_pivot, user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
7349- STBIR_MEMCPY( &info->vertical, vertical, sizeof( stbir__sampler ) );
7350-
7351- STBIR_PROFILE_BUILD_END( vertical );
7352- }
7353-
7354- // setup the vertical split ranges
7355- stbir__get_split_info( info->split_info, info->splits, info->vertical.scale_info.output_sub_size, info->vertical.filter_pixel_margin, info->vertical.scale_info.input_full_size, info->vertical.is_gather, info->vertical.contributors );
7356-
7357- // now we know precisely how many entries we need
7358- info->ring_buffer_num_entries = info->vertical.extent_info.widest;
7359-
7360- // we never need more ring buffer entries than the scanlines we're outputting
7361- if ( ( !info->vertical.is_gather ) && ( info->ring_buffer_num_entries > conservative_split_output_size ) )
7362- info->ring_buffer_num_entries = conservative_split_output_size;
7363- STBIR_ASSERT( info->ring_buffer_num_entries <= info->alloc_ring_buffer_num_entries );
7364- }
7365- #undef STBIR__NEXT_PTR
7366-
7367-
7368- // is this the first time through loop?
7369- if ( info == 0 )
7370- {
7371- alloced_total = ( 15 + (size_t)advance_mem );
7372- alloced = STBIR_MALLOC( alloced_total, user_data );
7373- if ( alloced == 0 )
7374- return 0;
7375- }
7376- else
7377- return info; // success
7378- }
7379-}
7380-
7381-static int stbir__perform_resize( stbir__info const * info, int split_start, int split_count )
7382-{
7383- stbir__per_split_info * split_info = info->split_info + split_start;
7384-
7385- STBIR_PROFILE_CLEAR_EXTRAS();
7386-
7387- STBIR_PROFILE_FIRST_START( looping );
7388- if (info->vertical.is_gather)
7389- stbir__vertical_gather_loop( info, split_info, split_count );
7390- else
7391- stbir__vertical_scatter_loop( info, split_info, split_count );
7392- STBIR_PROFILE_END( looping );
7393-
7394- return 1;
7395-}
7396-
7397-static void stbir__update_info_from_resize( stbir__info * info, STBIR_RESIZE * resize )
7398-{
7399- static stbir__decode_pixels_func * decode_simple[STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
7400- {
7401- /* 1ch-4ch */ stbir__decode_uint8_srgb, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear,
7402- };
7403-
7404- static stbir__decode_pixels_func * decode_alphas[STBIRI_AR-STBIRI_RGBA+1][STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
7405- {
7406- { /* RGBA */ stbir__decode_uint8_srgb4_linearalpha, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear },
7407- { /* BGRA */ stbir__decode_uint8_srgb4_linearalpha_BGRA, stbir__decode_uint8_srgb_BGRA, 0, stbir__decode_float_linear_BGRA, stbir__decode_half_float_linear_BGRA },
7408- { /* ARGB */ stbir__decode_uint8_srgb4_linearalpha_ARGB, stbir__decode_uint8_srgb_ARGB, 0, stbir__decode_float_linear_ARGB, stbir__decode_half_float_linear_ARGB },
7409- { /* ABGR */ stbir__decode_uint8_srgb4_linearalpha_ABGR, stbir__decode_uint8_srgb_ABGR, 0, stbir__decode_float_linear_ABGR, stbir__decode_half_float_linear_ABGR },
7410- { /* RA */ stbir__decode_uint8_srgb2_linearalpha, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear },
7411- { /* AR */ stbir__decode_uint8_srgb2_linearalpha_AR, stbir__decode_uint8_srgb_AR, 0, stbir__decode_float_linear_AR, stbir__decode_half_float_linear_AR },
7412- };
7413-
7414- static stbir__decode_pixels_func * decode_simple_scaled_or_not[2][2]=
7415- {
7416- { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear },
7417- };
7418-
7419- static stbir__decode_pixels_func * decode_alphas_scaled_or_not[STBIRI_AR-STBIRI_RGBA+1][2][2]=
7420- {
7421- { /* RGBA */ { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear } },
7422- { /* BGRA */ { stbir__decode_uint8_linear_scaled_BGRA, stbir__decode_uint8_linear_BGRA }, { stbir__decode_uint16_linear_scaled_BGRA, stbir__decode_uint16_linear_BGRA } },
7423- { /* ARGB */ { stbir__decode_uint8_linear_scaled_ARGB, stbir__decode_uint8_linear_ARGB }, { stbir__decode_uint16_linear_scaled_ARGB, stbir__decode_uint16_linear_ARGB } },
7424- { /* ABGR */ { stbir__decode_uint8_linear_scaled_ABGR, stbir__decode_uint8_linear_ABGR }, { stbir__decode_uint16_linear_scaled_ABGR, stbir__decode_uint16_linear_ABGR } },
7425- { /* RA */ { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear } },
7426- { /* AR */ { stbir__decode_uint8_linear_scaled_AR, stbir__decode_uint8_linear_AR }, { stbir__decode_uint16_linear_scaled_AR, stbir__decode_uint16_linear_AR } }
7427- };
7428-
7429- static stbir__encode_pixels_func * encode_simple[STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
7430- {
7431- /* 1ch-4ch */ stbir__encode_uint8_srgb, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear,
7432- };
7433-
7434- static stbir__encode_pixels_func * encode_alphas[STBIRI_AR-STBIRI_RGBA+1][STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
7435- {
7436- { /* RGBA */ stbir__encode_uint8_srgb4_linearalpha, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear },
7437- { /* BGRA */ stbir__encode_uint8_srgb4_linearalpha_BGRA, stbir__encode_uint8_srgb_BGRA, 0, stbir__encode_float_linear_BGRA, stbir__encode_half_float_linear_BGRA },
7438- { /* ARGB */ stbir__encode_uint8_srgb4_linearalpha_ARGB, stbir__encode_uint8_srgb_ARGB, 0, stbir__encode_float_linear_ARGB, stbir__encode_half_float_linear_ARGB },
7439- { /* ABGR */ stbir__encode_uint8_srgb4_linearalpha_ABGR, stbir__encode_uint8_srgb_ABGR, 0, stbir__encode_float_linear_ABGR, stbir__encode_half_float_linear_ABGR },
7440- { /* RA */ stbir__encode_uint8_srgb2_linearalpha, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear },
7441- { /* AR */ stbir__encode_uint8_srgb2_linearalpha_AR, stbir__encode_uint8_srgb_AR, 0, stbir__encode_float_linear_AR, stbir__encode_half_float_linear_AR }
7442- };
7443-
7444- static stbir__encode_pixels_func * encode_simple_scaled_or_not[2][2]=
7445- {
7446- { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear },
7447- };
7448-
7449- static stbir__encode_pixels_func * encode_alphas_scaled_or_not[STBIRI_AR-STBIRI_RGBA+1][2][2]=
7450- {
7451- { /* RGBA */ { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear } },
7452- { /* BGRA */ { stbir__encode_uint8_linear_scaled_BGRA, stbir__encode_uint8_linear_BGRA }, { stbir__encode_uint16_linear_scaled_BGRA, stbir__encode_uint16_linear_BGRA } },
7453- { /* ARGB */ { stbir__encode_uint8_linear_scaled_ARGB, stbir__encode_uint8_linear_ARGB }, { stbir__encode_uint16_linear_scaled_ARGB, stbir__encode_uint16_linear_ARGB } },
7454- { /* ABGR */ { stbir__encode_uint8_linear_scaled_ABGR, stbir__encode_uint8_linear_ABGR }, { stbir__encode_uint16_linear_scaled_ABGR, stbir__encode_uint16_linear_ABGR } },
7455- { /* RA */ { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear } },
7456- { /* AR */ { stbir__encode_uint8_linear_scaled_AR, stbir__encode_uint8_linear_AR }, { stbir__encode_uint16_linear_scaled_AR, stbir__encode_uint16_linear_AR } }
7457- };
7458-
7459- stbir__decode_pixels_func * decode_pixels = 0;
7460- stbir__encode_pixels_func * encode_pixels = 0;
7461- stbir_datatype input_type, output_type;
7462-
7463- input_type = resize->input_data_type;
7464- output_type = resize->output_data_type;
7465- info->input_data = resize->input_pixels;
7466- info->input_stride_bytes = resize->input_stride_in_bytes;
7467- info->output_stride_bytes = resize->output_stride_in_bytes;
7468-
7469- // if we're completely point sampling, then we can turn off SRGB
7470- if ( ( info->horizontal.filter_enum == STBIR_FILTER_POINT_SAMPLE ) && ( info->vertical.filter_enum == STBIR_FILTER_POINT_SAMPLE ) )
7471- {
7472- if ( ( ( input_type == STBIR_TYPE_UINT8_SRGB ) || ( input_type == STBIR_TYPE_UINT8_SRGB_ALPHA ) ) &&
7473- ( ( output_type == STBIR_TYPE_UINT8_SRGB ) || ( output_type == STBIR_TYPE_UINT8_SRGB_ALPHA ) ) )
7474- {
7475- input_type = STBIR_TYPE_UINT8;
7476- output_type = STBIR_TYPE_UINT8;
7477- }
7478- }
7479-
7480- // recalc the output and input strides
7481- if ( info->input_stride_bytes == 0 )
7482- info->input_stride_bytes = info->channels * info->horizontal.scale_info.input_full_size * stbir__type_size[input_type];
7483-
7484- if ( info->output_stride_bytes == 0 )
7485- info->output_stride_bytes = info->channels * info->horizontal.scale_info.output_sub_size * stbir__type_size[output_type];
7486-
7487- // calc offset
7488- info->output_data = ( (char*) resize->output_pixels ) + ( (size_t) info->offset_y * (size_t) resize->output_stride_in_bytes ) + ( info->offset_x * info->channels * stbir__type_size[output_type] );
7489-
7490- info->in_pixels_cb = resize->input_cb;
7491- info->user_data = resize->user_data;
7492- info->out_pixels_cb = resize->output_cb;
7493-
7494- // setup the input format converters
7495- if ( ( input_type == STBIR_TYPE_UINT8 ) || ( input_type == STBIR_TYPE_UINT16 ) )
7496- {
7497- int non_scaled = 0;
7498-
7499- // check if we can run unscaled - 0-255.0/0-65535.0 instead of 0-1.0 (which is a tiny bit faster when doing linear 8->8 or 16->16)
7500- if ( ( !info->alpha_weight ) && ( !info->alpha_unweight ) ) // don't short circuit when alpha weighting (get everything to 0-1.0 as usual)
7501- if ( ( ( input_type == STBIR_TYPE_UINT8 ) && ( output_type == STBIR_TYPE_UINT8 ) ) || ( ( input_type == STBIR_TYPE_UINT16 ) && ( output_type == STBIR_TYPE_UINT16 ) ) )
7502- non_scaled = 1;
7503-
7504- if ( info->input_pixel_layout_internal <= STBIRI_4CHANNEL )
7505- decode_pixels = decode_simple_scaled_or_not[ input_type == STBIR_TYPE_UINT16 ][ non_scaled ];
7506- else
7507- decode_pixels = decode_alphas_scaled_or_not[ ( info->input_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ input_type == STBIR_TYPE_UINT16 ][ non_scaled ];
7508- }
7509- else
7510- {
7511- if ( info->input_pixel_layout_internal <= STBIRI_4CHANNEL )
7512- decode_pixels = decode_simple[ input_type - STBIR_TYPE_UINT8_SRGB ];
7513- else
7514- decode_pixels = decode_alphas[ ( info->input_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ input_type - STBIR_TYPE_UINT8_SRGB ];
7515- }
7516-
7517- // setup the output format converters
7518- if ( ( output_type == STBIR_TYPE_UINT8 ) || ( output_type == STBIR_TYPE_UINT16 ) )
7519- {
7520- int non_scaled = 0;
7521-
7522- // check if we can run unscaled - 0-255.0/0-65535.0 instead of 0-1.0 (which is a tiny bit faster when doing linear 8->8 or 16->16)
7523- if ( ( !info->alpha_weight ) && ( !info->alpha_unweight ) ) // don't short circuit when alpha weighting (get everything to 0-1.0 as usual)
7524- if ( ( ( input_type == STBIR_TYPE_UINT8 ) && ( output_type == STBIR_TYPE_UINT8 ) ) || ( ( input_type == STBIR_TYPE_UINT16 ) && ( output_type == STBIR_TYPE_UINT16 ) ) )
7525- non_scaled = 1;
7526-
7527- if ( info->output_pixel_layout_internal <= STBIRI_4CHANNEL )
7528- encode_pixels = encode_simple_scaled_or_not[ output_type == STBIR_TYPE_UINT16 ][ non_scaled ];
7529- else
7530- encode_pixels = encode_alphas_scaled_or_not[ ( info->output_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ output_type == STBIR_TYPE_UINT16 ][ non_scaled ];
7531- }
7532- else
7533- {
7534- if ( info->output_pixel_layout_internal <= STBIRI_4CHANNEL )
7535- encode_pixels = encode_simple[ output_type - STBIR_TYPE_UINT8_SRGB ];
7536- else
7537- encode_pixels = encode_alphas[ ( info->output_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ output_type - STBIR_TYPE_UINT8_SRGB ];
7538- }
7539-
7540- info->input_type = input_type;
7541- info->output_type = output_type;
7542- info->decode_pixels = decode_pixels;
7543- info->encode_pixels = encode_pixels;
7544-}
7545-
7546-static void stbir__clip( int * outx, int * outsubw, int outw, double * u0, double * u1 )
7547-{
7548- double per, adj;
7549- int over;
7550-
7551- // do left/top edge
7552- if ( *outx < 0 )
7553- {
7554- per = ( (double)*outx ) / ( (double)*outsubw ); // is negative
7555- adj = per * ( *u1 - *u0 );
7556- *u0 -= adj; // increases u0
7557- *outx = 0;
7558- }
7559-
7560- // do right/bot edge
7561- over = outw - ( *outx + *outsubw );
7562- if ( over < 0 )
7563- {
7564- per = ( (double)over ) / ( (double)*outsubw ); // is negative
7565- adj = per * ( *u1 - *u0 );
7566- *u1 += adj; // decrease u1
7567- *outsubw = outw - *outx;
7568- }
7569-}
7570-
7571-// converts a double to a rational that has less than one float bit of error (returns 0 if unable to do so)
7572-static int stbir__double_to_rational(double f, stbir_uint32 limit, stbir_uint32 *numer, stbir_uint32 *denom, int limit_denom ) // limit_denom (1) or limit numer (0)
7573-{
7574- double err;
7575- stbir_uint64 top, bot;
7576- stbir_uint64 numer_last = 0;
7577- stbir_uint64 denom_last = 1;
7578- stbir_uint64 numer_estimate = 1;
7579- stbir_uint64 denom_estimate = 0;
7580-
7581- // scale to past float error range
7582- top = (stbir_uint64)( f * (double)(1 << 25) );
7583- bot = 1 << 25;
7584-
7585- // keep refining, but usually stops in a few loops - usually 5 for bad cases
7586- for(;;)
7587- {
7588- stbir_uint64 est, temp;
7589-
7590- // hit limit, break out and do best full range estimate
7591- if ( ( ( limit_denom ) ? denom_estimate : numer_estimate ) >= limit )
7592- break;
7593-
7594- // is the current error less than 1 bit of a float? if so, we're done
7595- if ( denom_estimate )
7596- {
7597- err = ( (double)numer_estimate / (double)denom_estimate ) - f;
7598- if ( err < 0.0 ) err = -err;
7599- if ( err < ( 1.0 / (double)(1<<24) ) )
7600- {
7601- // yup, found it
7602- *numer = (stbir_uint32) numer_estimate;
7603- *denom = (stbir_uint32) denom_estimate;
7604- return 1;
7605- }
7606- }
7607-
7608- // no more refinement bits left? break out and do full range estimate
7609- if ( bot == 0 )
7610- break;
7611-
7612- // gcd the estimate bits
7613- est = top / bot;
7614- temp = top % bot;
7615- top = bot;
7616- bot = temp;
7617-
7618- // move remainders
7619- temp = est * denom_estimate + denom_last;
7620- denom_last = denom_estimate;
7621- denom_estimate = temp;
7622-
7623- // move remainders
7624- temp = est * numer_estimate + numer_last;
7625- numer_last = numer_estimate;
7626- numer_estimate = temp;
7627- }
7628-
7629- // we didn't find anything good enough for float, use a full range estimate
7630- if ( limit_denom )
7631- {
7632- numer_estimate= (stbir_uint64)( f * (double)limit + 0.5 );
7633- denom_estimate = limit;
7634- }
7635- else
7636- {
7637- numer_estimate = limit;
7638- denom_estimate = (stbir_uint64)( ( (double)limit / f ) + 0.5 );
7639- }
7640-
7641- *numer = (stbir_uint32) numer_estimate;
7642- *denom = (stbir_uint32) denom_estimate;
7643-
7644- err = ( denom_estimate ) ? ( ( (double)(stbir_uint32)numer_estimate / (double)(stbir_uint32)denom_estimate ) - f ) : 1.0;
7645- if ( err < 0.0 ) err = -err;
7646- return ( err < ( 1.0 / (double)(1<<24) ) ) ? 1 : 0;
7647-}
7648-
7649-static int stbir__calculate_region_transform( stbir__scale_info * scale_info, int output_full_range, int * output_offset, int output_sub_range, int input_full_range, double input_s0, double input_s1 )
7650-{
7651- double output_range, input_range, output_s, input_s, ratio, scale;
7652-
7653- input_s = input_s1 - input_s0;
7654-
7655- // null area
7656- if ( ( output_full_range == 0 ) || ( input_full_range == 0 ) ||
7657- ( output_sub_range == 0 ) || ( input_s <= stbir__small_float ) )
7658- return 0;
7659-
7660- // are either of the ranges completely out of bounds?
7661- if ( ( *output_offset >= output_full_range ) || ( ( *output_offset + output_sub_range ) <= 0 ) || ( input_s0 >= (1.0f-stbir__small_float) ) || ( input_s1 <= stbir__small_float ) )
7662- return 0;
7663-
7664- output_range = (double)output_full_range;
7665- input_range = (double)input_full_range;
7666-
7667- output_s = ( (double)output_sub_range) / output_range;
7668-
7669- // figure out the scaling to use
7670- ratio = output_s / input_s;
7671-
7672- // save scale before clipping
7673- scale = ( output_range / input_range ) * ratio;
7674- scale_info->scale = (float)scale;
7675- scale_info->inv_scale = (float)( 1.0 / scale );
7676-
7677- // clip output area to left/right output edges (and adjust input area)
7678- stbir__clip( output_offset, &output_sub_range, output_full_range, &input_s0, &input_s1 );
7679-
7680- // recalc input area
7681- input_s = input_s1 - input_s0;
7682-
7683- // after clipping do we have zero input area?
7684- if ( input_s <= stbir__small_float )
7685- return 0;
7686-
7687- // calculate and store the starting source offsets in output pixel space
7688- scale_info->pixel_shift = (float) ( input_s0 * ratio * output_range );
7689-
7690- scale_info->scale_is_rational = stbir__double_to_rational( scale, ( scale <= 1.0 ) ? output_full_range : input_full_range, &scale_info->scale_numerator, &scale_info->scale_denominator, ( scale >= 1.0 ) );
7691-
7692- scale_info->input_full_size = input_full_range;
7693- scale_info->output_sub_size = output_sub_range;
7694-
7695- return 1;
7696-}
7697-
7698-
7699-static void stbir__init_and_set_layout( STBIR_RESIZE * resize, stbir_pixel_layout pixel_layout, stbir_datatype data_type )
7700-{
7701- resize->input_cb = 0;
7702- resize->output_cb = 0;
7703- resize->user_data = resize;
7704- resize->samplers = 0;
7705- resize->called_alloc = 0;
7706- resize->horizontal_filter = STBIR_FILTER_DEFAULT;
7707- resize->horizontal_filter_kernel = 0; resize->horizontal_filter_support = 0;
7708- resize->vertical_filter = STBIR_FILTER_DEFAULT;
7709- resize->vertical_filter_kernel = 0; resize->vertical_filter_support = 0;
7710- resize->horizontal_edge = STBIR_EDGE_CLAMP;
7711- resize->vertical_edge = STBIR_EDGE_CLAMP;
7712- resize->input_s0 = 0; resize->input_t0 = 0; resize->input_s1 = 1; resize->input_t1 = 1;
7713- resize->output_subx = 0; resize->output_suby = 0; resize->output_subw = resize->output_w; resize->output_subh = resize->output_h;
7714- resize->input_data_type = data_type;
7715- resize->output_data_type = data_type;
7716- resize->input_pixel_layout_public = pixel_layout;
7717- resize->output_pixel_layout_public = pixel_layout;
7718- resize->needs_rebuild = 1;
7719-}
7720-
7721-STBIRDEF void stbir_resize_init( STBIR_RESIZE * resize,
7722- const void *input_pixels, int input_w, int input_h, int input_stride_in_bytes, // stride can be zero
7723- void *output_pixels, int output_w, int output_h, int output_stride_in_bytes, // stride can be zero
7724- stbir_pixel_layout pixel_layout, stbir_datatype data_type )
7725-{
7726- resize->input_pixels = input_pixels;
7727- resize->input_w = input_w;
7728- resize->input_h = input_h;
7729- resize->input_stride_in_bytes = input_stride_in_bytes;
7730- resize->output_pixels = output_pixels;
7731- resize->output_w = output_w;
7732- resize->output_h = output_h;
7733- resize->output_stride_in_bytes = output_stride_in_bytes;
7734- resize->fast_alpha = 0;
7735-
7736- stbir__init_and_set_layout( resize, pixel_layout, data_type );
7737-}
7738-
7739-// You can update parameters any time after resize_init
7740-STBIRDEF void stbir_set_datatypes( STBIR_RESIZE * resize, stbir_datatype input_type, stbir_datatype output_type ) // by default, datatype from resize_init
7741-{
7742- resize->input_data_type = input_type;
7743- resize->output_data_type = output_type;
7744- if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
7745- stbir__update_info_from_resize( resize->samplers, resize );
7746-}
7747-
7748-STBIRDEF void stbir_set_pixel_callbacks( STBIR_RESIZE * resize, stbir_input_callback * input_cb, stbir_output_callback * output_cb ) // no callbacks by default
7749-{
7750- resize->input_cb = input_cb;
7751- resize->output_cb = output_cb;
7752-
7753- if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
7754- {
7755- resize->samplers->in_pixels_cb = input_cb;
7756- resize->samplers->out_pixels_cb = output_cb;
7757- }
7758-}
7759-
7760-STBIRDEF void stbir_set_user_data( STBIR_RESIZE * resize, void * user_data ) // pass back STBIR_RESIZE* by default
7761-{
7762- resize->user_data = user_data;
7763- if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
7764- resize->samplers->user_data = user_data;
7765-}
7766-
7767-STBIRDEF void stbir_set_buffer_ptrs( STBIR_RESIZE * resize, const void * input_pixels, int input_stride_in_bytes, void * output_pixels, int output_stride_in_bytes )
7768-{
7769- resize->input_pixels = input_pixels;
7770- resize->input_stride_in_bytes = input_stride_in_bytes;
7771- resize->output_pixels = output_pixels;
7772- resize->output_stride_in_bytes = output_stride_in_bytes;
7773- if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
7774- stbir__update_info_from_resize( resize->samplers, resize );
7775-}
7776-
7777-
7778-STBIRDEF int stbir_set_edgemodes( STBIR_RESIZE * resize, stbir_edge horizontal_edge, stbir_edge vertical_edge ) // CLAMP by default
7779-{
7780- resize->horizontal_edge = horizontal_edge;
7781- resize->vertical_edge = vertical_edge;
7782- resize->needs_rebuild = 1;
7783- return 1;
7784-}
7785-
7786-STBIRDEF int stbir_set_filters( STBIR_RESIZE * resize, stbir_filter horizontal_filter, stbir_filter vertical_filter ) // STBIR_DEFAULT_FILTER_UPSAMPLE/DOWNSAMPLE by default
7787-{
7788- resize->horizontal_filter = horizontal_filter;
7789- resize->vertical_filter = vertical_filter;
7790- resize->needs_rebuild = 1;
7791- return 1;
7792-}
7793-
7794-STBIRDEF int stbir_set_filter_callbacks( STBIR_RESIZE * resize, stbir__kernel_callback * horizontal_filter, stbir__support_callback * horizontal_support, stbir__kernel_callback * vertical_filter, stbir__support_callback * vertical_support )
7795-{
7796- resize->horizontal_filter_kernel = horizontal_filter; resize->horizontal_filter_support = horizontal_support;
7797- resize->vertical_filter_kernel = vertical_filter; resize->vertical_filter_support = vertical_support;
7798- resize->needs_rebuild = 1;
7799- return 1;
7800-}
7801-
7802-STBIRDEF int stbir_set_pixel_layouts( STBIR_RESIZE * resize, stbir_pixel_layout input_pixel_layout, stbir_pixel_layout output_pixel_layout ) // sets new pixel layouts
7803-{
7804- resize->input_pixel_layout_public = input_pixel_layout;
7805- resize->output_pixel_layout_public = output_pixel_layout;
7806- resize->needs_rebuild = 1;
7807- return 1;
7808-}
7809-
7810-
7811-STBIRDEF int stbir_set_non_pm_alpha_speed_over_quality( STBIR_RESIZE * resize, int non_pma_alpha_speed_over_quality ) // sets alpha speed
7812-{
7813- resize->fast_alpha = non_pma_alpha_speed_over_quality;
7814- resize->needs_rebuild = 1;
7815- return 1;
7816-}
7817-
7818-STBIRDEF int stbir_set_input_subrect( STBIR_RESIZE * resize, double s0, double t0, double s1, double t1 ) // sets input region (full region by default)
7819-{
7820- resize->input_s0 = s0;
7821- resize->input_t0 = t0;
7822- resize->input_s1 = s1;
7823- resize->input_t1 = t1;
7824- resize->needs_rebuild = 1;
7825-
7826- // are we inbounds?
7827- if ( ( s1 < stbir__small_float ) || ( (s1-s0) < stbir__small_float ) ||
7828- ( t1 < stbir__small_float ) || ( (t1-t0) < stbir__small_float ) ||
7829- ( s0 > (1.0f-stbir__small_float) ) ||
7830- ( t0 > (1.0f-stbir__small_float) ) )
7831- return 0;
7832-
7833- return 1;
7834-}
7835-
7836-STBIRDEF int stbir_set_output_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ) // sets input region (full region by default)
7837-{
7838- resize->output_subx = subx;
7839- resize->output_suby = suby;
7840- resize->output_subw = subw;
7841- resize->output_subh = subh;
7842- resize->needs_rebuild = 1;
7843-
7844- // are we inbounds?
7845- if ( ( subx >= resize->output_w ) || ( ( subx + subw ) <= 0 ) || ( suby >= resize->output_h ) || ( ( suby + subh ) <= 0 ) || ( subw == 0 ) || ( subh == 0 ) )
7846- return 0;
7847-
7848- return 1;
7849-}
7850-
7851-STBIRDEF int stbir_set_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ) // sets both regions (full regions by default)
7852-{
7853- double s0, t0, s1, t1;
7854-
7855- s0 = ( (double)subx ) / ( (double)resize->output_w );
7856- t0 = ( (double)suby ) / ( (double)resize->output_h );
7857- s1 = ( (double)(subx+subw) ) / ( (double)resize->output_w );
7858- t1 = ( (double)(suby+subh) ) / ( (double)resize->output_h );
7859-
7860- resize->input_s0 = s0;
7861- resize->input_t0 = t0;
7862- resize->input_s1 = s1;
7863- resize->input_t1 = t1;
7864- resize->output_subx = subx;
7865- resize->output_suby = suby;
7866- resize->output_subw = subw;
7867- resize->output_subh = subh;
7868- resize->needs_rebuild = 1;
7869-
7870- // are we inbounds?
7871- if ( ( subx >= resize->output_w ) || ( ( subx + subw ) <= 0 ) || ( suby >= resize->output_h ) || ( ( suby + subh ) <= 0 ) || ( subw == 0 ) || ( subh == 0 ) )
7872- return 0;
7873-
7874- return 1;
7875-}
7876-
7877-static int stbir__perform_build( STBIR_RESIZE * resize, int splits )
7878-{
7879- stbir__contributors conservative = { 0, 0 };
7880- stbir__sampler horizontal, vertical;
7881- int new_output_subx, new_output_suby;
7882- stbir__info * out_info;
7883- #ifdef STBIR_PROFILE
7884- stbir__info profile_infod; // used to contain building profile info before everything is allocated
7885- stbir__info * profile_info = &profile_infod;
7886- #endif
7887-
7888- // have we already built the samplers?
7889- if ( resize->samplers )
7890- return 0;
7891-
7892- #define STBIR_RETURN_ERROR_AND_ASSERT( exp ) STBIR_ASSERT( !(exp) ); if (exp) return 0;
7893- STBIR_RETURN_ERROR_AND_ASSERT( (unsigned)resize->horizontal_filter >= STBIR_FILTER_OTHER)
7894- STBIR_RETURN_ERROR_AND_ASSERT( (unsigned)resize->vertical_filter >= STBIR_FILTER_OTHER)
7895- #undef STBIR_RETURN_ERROR_AND_ASSERT
7896-
7897- if ( splits <= 0 )
7898- return 0;
7899-
7900- STBIR_PROFILE_BUILD_FIRST_START( build );
7901-
7902- new_output_subx = resize->output_subx;
7903- new_output_suby = resize->output_suby;
7904-
7905- // do horizontal clip and scale calcs
7906- if ( !stbir__calculate_region_transform( &horizontal.scale_info, resize->output_w, &new_output_subx, resize->output_subw, resize->input_w, resize->input_s0, resize->input_s1 ) )
7907- return 0;
7908-
7909- // do vertical clip and scale calcs
7910- if ( !stbir__calculate_region_transform( &vertical.scale_info, resize->output_h, &new_output_suby, resize->output_subh, resize->input_h, resize->input_t0, resize->input_t1 ) )
7911- return 0;
7912-
7913- // if nothing to do, just return
7914- if ( ( horizontal.scale_info.output_sub_size == 0 ) || ( vertical.scale_info.output_sub_size == 0 ) )
7915- return 0;
7916-
7917- stbir__set_sampler(&horizontal, resize->horizontal_filter, resize->horizontal_filter_kernel, resize->horizontal_filter_support, resize->horizontal_edge, &horizontal.scale_info, 1, resize->user_data );
7918- stbir__get_conservative_extents( &horizontal, &conservative, resize->user_data );
7919- stbir__set_sampler(&vertical, resize->vertical_filter, resize->vertical_filter_kernel, resize->vertical_filter_support, resize->vertical_edge, &vertical.scale_info, 0, resize->user_data );
7920-
7921- if ( ( vertical.scale_info.output_sub_size / splits ) < STBIR_FORCE_MINIMUM_SCANLINES_FOR_SPLITS ) // each split should be a minimum of 4 scanlines (handwavey choice)
7922- {
7923- splits = vertical.scale_info.output_sub_size / STBIR_FORCE_MINIMUM_SCANLINES_FOR_SPLITS;
7924- if ( splits == 0 ) splits = 1;
7925- }
7926-
7927- STBIR_PROFILE_BUILD_START( alloc );
7928- out_info = stbir__alloc_internal_mem_and_build_samplers( &horizontal, &vertical, &conservative, resize->input_pixel_layout_public, resize->output_pixel_layout_public, splits, new_output_subx, new_output_suby, resize->fast_alpha, resize->user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
7929- STBIR_PROFILE_BUILD_END( alloc );
7930- STBIR_PROFILE_BUILD_END( build );
7931-
7932- if ( out_info )
7933- {
7934- resize->splits = splits;
7935- resize->samplers = out_info;
7936- resize->needs_rebuild = 0;
7937- #ifdef STBIR_PROFILE
7938- STBIR_MEMCPY( &out_info->profile, &profile_infod.profile, sizeof( out_info->profile ) );
7939- #endif
7940-
7941- // update anything that can be changed without recalcing samplers
7942- stbir__update_info_from_resize( out_info, resize );
7943-
7944- return splits;
7945- }
7946-
7947- return 0;
7948-}
7949-
7950-STBIRDEF void stbir_free_samplers( STBIR_RESIZE * resize )
7951-{
7952- if ( resize->samplers )
7953- {
7954- stbir__free_internal_mem( resize->samplers );
7955- resize->samplers = 0;
7956- resize->called_alloc = 0;
7957- }
7958-}
7959-
7960-STBIRDEF int stbir_build_samplers_with_splits( STBIR_RESIZE * resize, int splits )
7961-{
7962- if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
7963- {
7964- if ( resize->samplers )
7965- stbir_free_samplers( resize );
7966-
7967- resize->called_alloc = 1;
7968- return stbir__perform_build( resize, splits );
7969- }
7970-
7971- STBIR_PROFILE_BUILD_CLEAR( resize->samplers );
7972-
7973- return 1;
7974-}
7975-
7976-STBIRDEF int stbir_build_samplers( STBIR_RESIZE * resize )
7977-{
7978- return stbir_build_samplers_with_splits( resize, 1 );
7979-}
7980-
7981-STBIRDEF int stbir_resize_extended( STBIR_RESIZE * resize )
7982-{
7983- int result;
7984-
7985- if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
7986- {
7987- int alloc_state = resize->called_alloc; // remember allocated state
7988-
7989- if ( resize->samplers )
7990- {
7991- stbir__free_internal_mem( resize->samplers );
7992- resize->samplers = 0;
7993- }
7994-
7995- if ( !stbir_build_samplers( resize ) )
7996- return 0;
7997-
7998- resize->called_alloc = alloc_state;
7999-
8000- // if build_samplers succeeded (above), but there are no samplers set, then
8001- // the area to stretch into was zero pixels, so don't do anything and return
8002- // success
8003- if ( resize->samplers == 0 )
8004- return 1;
8005- }
8006- else
8007- {
8008- // didn't build anything - clear it
8009- STBIR_PROFILE_BUILD_CLEAR( resize->samplers );
8010- }
8011-
8012- // do resize
8013- result = stbir__perform_resize( resize->samplers, 0, resize->splits );
8014-
8015- // if we alloced, then free
8016- if ( !resize->called_alloc )
8017- {
8018- stbir_free_samplers( resize );
8019- resize->samplers = 0;
8020- }
8021-
8022- return result;
8023-}
8024-
8025-STBIRDEF int stbir_resize_extended_split( STBIR_RESIZE * resize, int split_start, int split_count )
8026-{
8027- STBIR_ASSERT( resize->samplers );
8028-
8029- // if we're just doing the whole thing, call full
8030- if ( ( split_start == -1 ) || ( ( split_start == 0 ) && ( split_count == resize->splits ) ) )
8031- return stbir_resize_extended( resize );
8032-
8033- // you **must** build samplers first when using split resize
8034- if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
8035- return 0;
8036-
8037- if ( ( split_start >= resize->splits ) || ( split_start < 0 ) || ( ( split_start + split_count ) > resize->splits ) || ( split_count <= 0 ) )
8038- return 0;
8039-
8040- // do resize
8041- return stbir__perform_resize( resize->samplers, split_start, split_count );
8042-}
8043-
8044-
8045-static void * stbir_quick_resize_helper( const void *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
8046- void *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
8047- stbir_pixel_layout pixel_layout, stbir_datatype data_type, stbir_edge edge, stbir_filter filter )
8048-{
8049- STBIR_RESIZE resize;
8050- int scanline_output_in_bytes;
8051- int positive_output_stride_in_bytes;
8052- void * start_ptr;
8053- void * free_ptr;
8054-
8055- scanline_output_in_bytes = output_w * stbir__type_size[ data_type ] * stbir__pixel_channels[ stbir__pixel_layout_convert_public_to_internal[ pixel_layout ] ];
8056- if ( scanline_output_in_bytes == 0 )
8057- return 0;
8058-
8059- // if zero stride, use scanline output
8060- if ( output_stride_in_bytes == 0 )
8061- output_stride_in_bytes = scanline_output_in_bytes;
8062-
8063- // abs value for inverted images (negative pitches)
8064- positive_output_stride_in_bytes = output_stride_in_bytes;
8065- if ( positive_output_stride_in_bytes < 0 )
8066- positive_output_stride_in_bytes = -positive_output_stride_in_bytes;
8067-
8068- // is the requested stride smaller than the scanline output? if so, just fail
8069- if ( positive_output_stride_in_bytes < scanline_output_in_bytes )
8070- return 0;
8071-
8072- start_ptr = output_pixels;
8073- free_ptr = 0; // no free pointer, since they passed buffer to use
8074-
8075- // did they pass a zero for the dest? if so, allocate the buffer
8076- if ( output_pixels == 0 )
8077- {
8078- size_t size;
8079- char * ptr;
8080-
8081- size = (size_t)positive_output_stride_in_bytes * (size_t)output_h;
8082- if ( size == 0 )
8083- return 0;
8084-
8085- ptr = (char*) STBIR_MALLOC( size, 0 );
8086- if ( ptr == 0 )
8087- return 0;
8088-
8089- free_ptr = ptr;
8090-
8091- // point at the last scanline, if they requested a flipped image
8092- if ( output_stride_in_bytes < 0 )
8093- start_ptr = ptr + ( (size_t)positive_output_stride_in_bytes * (size_t)( output_h - 1 ) );
8094- else
8095- start_ptr = ptr;
8096- }
8097-
8098- // ok, now do the resize
8099- stbir_resize_init( &resize,
8100- input_pixels, input_w, input_h, input_stride_in_bytes,
8101- start_ptr, output_w, output_h, output_stride_in_bytes,
8102- pixel_layout, data_type );
8103-
8104- resize.horizontal_edge = edge;
8105- resize.vertical_edge = edge;
8106- resize.horizontal_filter = filter;
8107- resize.vertical_filter = filter;
8108-
8109- if ( !stbir_resize_extended( &resize ) )
8110- {
8111- if ( free_ptr )
8112- STBIR_FREE( free_ptr, 0 );
8113- return 0;
8114- }
8115-
8116- return (free_ptr) ? free_ptr : start_ptr;
8117-}
8118-
8119-
8120-
8121-STBIRDEF unsigned char * stbir_resize_uint8_linear( const unsigned char *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
8122- unsigned char *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
8123- stbir_pixel_layout pixel_layout )
8124-{
8125- return (unsigned char *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
8126- output_pixels, output_w, output_h, output_stride_in_bytes,
8127- pixel_layout, STBIR_TYPE_UINT8, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
8128-}
8129-
8130-STBIRDEF unsigned char * stbir_resize_uint8_srgb( const unsigned char *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
8131- unsigned char *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
8132- stbir_pixel_layout pixel_layout )
8133-{
8134- return (unsigned char *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
8135- output_pixels, output_w, output_h, output_stride_in_bytes,
8136- pixel_layout, STBIR_TYPE_UINT8_SRGB, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
8137-}
8138-
8139-
8140-STBIRDEF float * stbir_resize_float_linear( const float *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
8141- float *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
8142- stbir_pixel_layout pixel_layout )
8143-{
8144- return (float *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
8145- output_pixels, output_w, output_h, output_stride_in_bytes,
8146- pixel_layout, STBIR_TYPE_FLOAT, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
8147-}
8148-
8149-
8150-STBIRDEF void * stbir_resize( const void *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
8151- void *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
8152- stbir_pixel_layout pixel_layout, stbir_datatype data_type,
8153- stbir_edge edge, stbir_filter filter )
8154-{
8155- return (void *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
8156- output_pixels, output_w, output_h, output_stride_in_bytes,
8157- pixel_layout, data_type, edge, filter );
8158-}
8159-
8160-#ifdef STBIR_PROFILE
8161-
8162-STBIRDEF void stbir_resize_build_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize )
8163-{
8164- static char const * bdescriptions[6] = { "Building", "Allocating", "Horizontal sampler", "Vertical sampler", "Coefficient cleanup", "Coefficient pivot" } ;
8165- stbir__info* samp = resize->samplers;
8166- int i;
8167-
8168- typedef int testa[ (STBIR__ARRAY_SIZE( bdescriptions ) == (STBIR__ARRAY_SIZE( samp->profile.array )-1) )?1:-1];
8169- typedef int testb[ (sizeof( samp->profile.array ) == (sizeof(samp->profile.named)) )?1:-1];
8170- typedef int testc[ (sizeof( info->clocks ) >= (sizeof(samp->profile.named)) )?1:-1];
8171-
8172- for( i = 0 ; i < STBIR__ARRAY_SIZE( bdescriptions ) ; i++)
8173- info->clocks[i] = samp->profile.array[i+1];
8174-
8175- info->total_clocks = samp->profile.named.total;
8176- info->descriptions = bdescriptions;
8177- info->count = STBIR__ARRAY_SIZE( bdescriptions );
8178-}
8179-
8180-STBIRDEF void stbir_resize_split_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize, int split_start, int split_count )
8181-{
8182- static char const * descriptions[7] = { "Looping", "Vertical sampling", "Horizontal sampling", "Scanline input", "Scanline output", "Alpha weighting", "Alpha unweighting" };
8183- stbir__per_split_info * split_info;
8184- int s, i;
8185-
8186- typedef int testa[ (STBIR__ARRAY_SIZE( descriptions ) == (STBIR__ARRAY_SIZE( split_info->profile.array )-1) )?1:-1];
8187- typedef int testb[ (sizeof( split_info->profile.array ) == (sizeof(split_info->profile.named)) )?1:-1];
8188- typedef int testc[ (sizeof( info->clocks ) >= (sizeof(split_info->profile.named)) )?1:-1];
8189-
8190- if ( split_start == -1 )
8191- {
8192- split_start = 0;
8193- split_count = resize->samplers->splits;
8194- }
8195-
8196- if ( ( split_start >= resize->splits ) || ( split_start < 0 ) || ( ( split_start + split_count ) > resize->splits ) || ( split_count <= 0 ) )
8197- {
8198- info->total_clocks = 0;
8199- info->descriptions = 0;
8200- info->count = 0;
8201- return;
8202- }
8203-
8204- split_info = resize->samplers->split_info + split_start;
8205-
8206- // sum up the profile from all the splits
8207- for( i = 0 ; i < STBIR__ARRAY_SIZE( descriptions ) ; i++ )
8208- {
8209- stbir_uint64 sum = 0;
8210- for( s = 0 ; s < split_count ; s++ )
8211- sum += split_info[s].profile.array[i+1];
8212- info->clocks[i] = sum;
8213- }
8214-
8215- info->total_clocks = split_info->profile.named.total;
8216- info->descriptions = descriptions;
8217- info->count = STBIR__ARRAY_SIZE( descriptions );
8218-}
8219-
8220-STBIRDEF void stbir_resize_extended_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize )
8221-{
8222- stbir_resize_split_profile_info( info, resize, -1, 0 );
8223-}
8224-
8225-#endif // STBIR_PROFILE
8226-
8227-#undef STBIR_BGR
8228-#undef STBIR_1CHANNEL
8229-#undef STBIR_2CHANNEL
8230-#undef STBIR_RGB
8231-#undef STBIR_RGBA
8232-#undef STBIR_4CHANNEL
8233-#undef STBIR_BGRA
8234-#undef STBIR_ARGB
8235-#undef STBIR_ABGR
8236-#undef STBIR_RA
8237-#undef STBIR_AR
8238-#undef STBIR_RGBA_PM
8239-#undef STBIR_BGRA_PM
8240-#undef STBIR_ARGB_PM
8241-#undef STBIR_ABGR_PM
8242-#undef STBIR_RA_PM
8243-#undef STBIR_AR_PM
8244-
8245-#endif // STB_IMAGE_RESIZE_IMPLEMENTATION
8246-
8247-#else // STB_IMAGE_RESIZE_HORIZONTALS&STB_IMAGE_RESIZE_DO_VERTICALS
8248-
8249-// we reinclude the header file to define all the horizontal functions
8250-// specializing each function for the number of coeffs is 20-40% faster *OVERALL*
8251-
8252-// by including the header file again this way, we can still debug the functions
8253-
8254-#define STBIR_strs_join2( start, mid, end ) start##mid##end
8255-#define STBIR_strs_join1( start, mid, end ) STBIR_strs_join2( start, mid, end )
8256-
8257-#define STBIR_strs_join24( start, mid1, mid2, end ) start##mid1##mid2##end
8258-#define STBIR_strs_join14( start, mid1, mid2, end ) STBIR_strs_join24( start, mid1, mid2, end )
8259-
8260-#ifdef STB_IMAGE_RESIZE_DO_CODERS
8261-
8262-#ifdef stbir__decode_suffix
8263-#define STBIR__CODER_NAME( name ) STBIR_strs_join1( name, _, stbir__decode_suffix )
8264-#else
8265-#define STBIR__CODER_NAME( name ) name
8266-#endif
8267-
8268-#ifdef stbir__decode_swizzle
8269-#define stbir__decode_simdf8_flip(reg) STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( stbir__simdf8_0123to,stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3),stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3)(reg, reg)
8270-#define stbir__decode_simdf4_flip(reg) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf_0123to,stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3)(reg, reg)
8271-#define stbir__encode_simdf8_unflip(reg) STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( stbir__simdf8_0123to,stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3),stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3)(reg, reg)
8272-#define stbir__encode_simdf4_unflip(reg) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf_0123to,stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3)(reg, reg)
8273-#else
8274-#define stbir__decode_order0 0
8275-#define stbir__decode_order1 1
8276-#define stbir__decode_order2 2
8277-#define stbir__decode_order3 3
8278-#define stbir__encode_order0 0
8279-#define stbir__encode_order1 1
8280-#define stbir__encode_order2 2
8281-#define stbir__encode_order3 3
8282-#define stbir__decode_simdf8_flip(reg)
8283-#define stbir__decode_simdf4_flip(reg)
8284-#define stbir__encode_simdf8_unflip(reg)
8285-#define stbir__encode_simdf4_unflip(reg)
8286-#endif
8287-
8288-#ifdef STBIR_SIMD8
8289-#define stbir__encode_simdfX_unflip stbir__encode_simdf8_unflip
8290-#else
8291-#define stbir__encode_simdfX_unflip stbir__encode_simdf4_unflip
8292-#endif
8293-
8294-static float * STBIR__CODER_NAME( stbir__decode_uint8_linear_scaled )( float * decodep, int width_times_channels, void const * inputp )
8295-{
8296- float STBIR_STREAMOUT_PTR( * ) decode = decodep;
8297- float * decode_end = (float*) decode + width_times_channels;
8298- unsigned char const * input = (unsigned char const*)inputp;
8299-
8300- #ifdef STBIR_SIMD
8301- unsigned char const * end_input_m16 = input + width_times_channels - 16;
8302- if ( width_times_channels >= 16 )
8303- {
8304- decode_end -= 16;
8305- STBIR_NO_UNROLL_LOOP_START_INF_FOR
8306- for(;;)
8307- {
8308- #ifdef STBIR_SIMD8
8309- stbir__simdi i; stbir__simdi8 o0,o1;
8310- stbir__simdf8 of0, of1;
8311- STBIR_NO_UNROLL(decode);
8312- stbir__simdi_load( i, input );
8313- stbir__simdi8_expand_u8_to_u32( o0, o1, i );
8314- stbir__simdi8_convert_i32_to_float( of0, o0 );
8315- stbir__simdi8_convert_i32_to_float( of1, o1 );
8316- stbir__simdf8_mult( of0, of0, STBIR_max_uint8_as_float_inverted8);
8317- stbir__simdf8_mult( of1, of1, STBIR_max_uint8_as_float_inverted8);
8318- stbir__decode_simdf8_flip( of0 );
8319- stbir__decode_simdf8_flip( of1 );
8320- stbir__simdf8_store( decode + 0, of0 );
8321- stbir__simdf8_store( decode + 8, of1 );
8322- #else
8323- stbir__simdi i, o0, o1, o2, o3;
8324- stbir__simdf of0, of1, of2, of3;
8325- STBIR_NO_UNROLL(decode);
8326- stbir__simdi_load( i, input );
8327- stbir__simdi_expand_u8_to_u32( o0,o1,o2,o3,i);
8328- stbir__simdi_convert_i32_to_float( of0, o0 );
8329- stbir__simdi_convert_i32_to_float( of1, o1 );
8330- stbir__simdi_convert_i32_to_float( of2, o2 );
8331- stbir__simdi_convert_i32_to_float( of3, o3 );
8332- stbir__simdf_mult( of0, of0, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
8333- stbir__simdf_mult( of1, of1, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
8334- stbir__simdf_mult( of2, of2, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
8335- stbir__simdf_mult( of3, of3, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
8336- stbir__decode_simdf4_flip( of0 );
8337- stbir__decode_simdf4_flip( of1 );
8338- stbir__decode_simdf4_flip( of2 );
8339- stbir__decode_simdf4_flip( of3 );
8340- stbir__simdf_store( decode + 0, of0 );
8341- stbir__simdf_store( decode + 4, of1 );
8342- stbir__simdf_store( decode + 8, of2 );
8343- stbir__simdf_store( decode + 12, of3 );
8344- #endif
8345- decode += 16;
8346- input += 16;
8347- if ( decode <= decode_end )
8348- continue;
8349- if ( decode == ( decode_end + 16 ) )
8350- break;
8351- decode = decode_end; // backup and do last couple
8352- input = end_input_m16;
8353- }
8354- return decode_end + 16;
8355- }
8356- #endif
8357-
8358- // try to do blocks of 4 when you can
8359- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8360- decode += 4;
8361- STBIR_SIMD_NO_UNROLL_LOOP_START
8362- while( decode <= decode_end )
8363- {
8364- STBIR_SIMD_NO_UNROLL(decode);
8365- decode[0-4] = ((float)(input[stbir__decode_order0])) * stbir__max_uint8_as_float_inverted;
8366- decode[1-4] = ((float)(input[stbir__decode_order1])) * stbir__max_uint8_as_float_inverted;
8367- decode[2-4] = ((float)(input[stbir__decode_order2])) * stbir__max_uint8_as_float_inverted;
8368- decode[3-4] = ((float)(input[stbir__decode_order3])) * stbir__max_uint8_as_float_inverted;
8369- decode += 4;
8370- input += 4;
8371- }
8372- decode -= 4;
8373- #endif
8374-
8375- // do the remnants
8376- #if stbir__coder_min_num < 4
8377- STBIR_NO_UNROLL_LOOP_START
8378- while( decode < decode_end )
8379- {
8380- STBIR_NO_UNROLL(decode);
8381- decode[0] = ((float)(input[stbir__decode_order0])) * stbir__max_uint8_as_float_inverted;
8382- #if stbir__coder_min_num >= 2
8383- decode[1] = ((float)(input[stbir__decode_order1])) * stbir__max_uint8_as_float_inverted;
8384- #endif
8385- #if stbir__coder_min_num >= 3
8386- decode[2] = ((float)(input[stbir__decode_order2])) * stbir__max_uint8_as_float_inverted;
8387- #endif
8388- decode += stbir__coder_min_num;
8389- input += stbir__coder_min_num;
8390- }
8391- #endif
8392-
8393- return decode_end;
8394-}
8395-
8396-static void STBIR__CODER_NAME( stbir__encode_uint8_linear_scaled )( void * outputp, int width_times_channels, float const * encode )
8397-{
8398- unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char *) outputp;
8399- unsigned char * end_output = ( (unsigned char *) output ) + width_times_channels;
8400-
8401- #ifdef STBIR_SIMD
8402- if ( width_times_channels >= stbir__simdfX_float_count*2 )
8403- {
8404- float const * end_encode_m8 = encode + width_times_channels - stbir__simdfX_float_count*2;
8405- end_output -= stbir__simdfX_float_count*2;
8406- STBIR_NO_UNROLL_LOOP_START_INF_FOR
8407- for(;;)
8408- {
8409- stbir__simdfX e0, e1;
8410- stbir__simdi i;
8411- STBIR_SIMD_NO_UNROLL(encode);
8412- stbir__simdfX_madd_mem( e0, STBIR_simd_point5X, STBIR_max_uint8_as_floatX, encode );
8413- stbir__simdfX_madd_mem( e1, STBIR_simd_point5X, STBIR_max_uint8_as_floatX, encode+stbir__simdfX_float_count );
8414- stbir__encode_simdfX_unflip( e0 );
8415- stbir__encode_simdfX_unflip( e1 );
8416- #ifdef STBIR_SIMD8
8417- stbir__simdf8_pack_to_16bytes( i, e0, e1 );
8418- stbir__simdi_store( output, i );
8419- #else
8420- stbir__simdf_pack_to_8bytes( i, e0, e1 );
8421- stbir__simdi_store2( output, i );
8422- #endif
8423- encode += stbir__simdfX_float_count*2;
8424- output += stbir__simdfX_float_count*2;
8425- if ( output <= end_output )
8426- continue;
8427- if ( output == ( end_output + stbir__simdfX_float_count*2 ) )
8428- break;
8429- output = end_output; // backup and do last couple
8430- encode = end_encode_m8;
8431- }
8432- return;
8433- }
8434-
8435- // try to do blocks of 4 when you can
8436- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8437- output += 4;
8438- STBIR_NO_UNROLL_LOOP_START
8439- while( output <= end_output )
8440- {
8441- stbir__simdf e0;
8442- stbir__simdi i0;
8443- STBIR_NO_UNROLL(encode);
8444- stbir__simdf_load( e0, encode );
8445- stbir__simdf_madd( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), e0 );
8446- stbir__encode_simdf4_unflip( e0 );
8447- stbir__simdf_pack_to_8bytes( i0, e0, e0 ); // only use first 4
8448- *(int*)(output-4) = stbir__simdi_to_int( i0 );
8449- output += 4;
8450- encode += 4;
8451- }
8452- output -= 4;
8453- #endif
8454-
8455- // do the remnants
8456- #if stbir__coder_min_num < 4
8457- STBIR_NO_UNROLL_LOOP_START
8458- while( output < end_output )
8459- {
8460- stbir__simdf e0;
8461- STBIR_NO_UNROLL(encode);
8462- stbir__simdf_madd1_mem( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order0 ); output[0] = stbir__simdf_convert_float_to_uint8( e0 );
8463- #if stbir__coder_min_num >= 2
8464- stbir__simdf_madd1_mem( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order1 ); output[1] = stbir__simdf_convert_float_to_uint8( e0 );
8465- #endif
8466- #if stbir__coder_min_num >= 3
8467- stbir__simdf_madd1_mem( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order2 ); output[2] = stbir__simdf_convert_float_to_uint8( e0 );
8468- #endif
8469- output += stbir__coder_min_num;
8470- encode += stbir__coder_min_num;
8471- }
8472- #endif
8473-
8474- #else
8475-
8476- // try to do blocks of 4 when you can
8477- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8478- output += 4;
8479- while( output <= end_output )
8480- {
8481- float f;
8482- f = encode[stbir__encode_order0] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[0-4] = (unsigned char)f;
8483- f = encode[stbir__encode_order1] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[1-4] = (unsigned char)f;
8484- f = encode[stbir__encode_order2] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[2-4] = (unsigned char)f;
8485- f = encode[stbir__encode_order3] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[3-4] = (unsigned char)f;
8486- output += 4;
8487- encode += 4;
8488- }
8489- output -= 4;
8490- #endif
8491-
8492- // do the remnants
8493- #if stbir__coder_min_num < 4
8494- STBIR_NO_UNROLL_LOOP_START
8495- while( output < end_output )
8496- {
8497- float f;
8498- STBIR_NO_UNROLL(encode);
8499- f = encode[stbir__encode_order0] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[0] = (unsigned char)f;
8500- #if stbir__coder_min_num >= 2
8501- f = encode[stbir__encode_order1] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[1] = (unsigned char)f;
8502- #endif
8503- #if stbir__coder_min_num >= 3
8504- f = encode[stbir__encode_order2] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[2] = (unsigned char)f;
8505- #endif
8506- output += stbir__coder_min_num;
8507- encode += stbir__coder_min_num;
8508- }
8509- #endif
8510- #endif
8511-}
8512-
8513-static float * STBIR__CODER_NAME(stbir__decode_uint8_linear)( float * decodep, int width_times_channels, void const * inputp )
8514-{
8515- float STBIR_STREAMOUT_PTR( * ) decode = decodep;
8516- float * decode_end = (float*) decode + width_times_channels;
8517- unsigned char const * input = (unsigned char const*)inputp;
8518-
8519- #ifdef STBIR_SIMD
8520- unsigned char const * end_input_m16 = input + width_times_channels - 16;
8521- if ( width_times_channels >= 16 )
8522- {
8523- decode_end -= 16;
8524- STBIR_NO_UNROLL_LOOP_START_INF_FOR
8525- for(;;)
8526- {
8527- #ifdef STBIR_SIMD8
8528- stbir__simdi i; stbir__simdi8 o0,o1;
8529- stbir__simdf8 of0, of1;
8530- STBIR_NO_UNROLL(decode);
8531- stbir__simdi_load( i, input );
8532- stbir__simdi8_expand_u8_to_u32( o0, o1, i );
8533- stbir__simdi8_convert_i32_to_float( of0, o0 );
8534- stbir__simdi8_convert_i32_to_float( of1, o1 );
8535- stbir__decode_simdf8_flip( of0 );
8536- stbir__decode_simdf8_flip( of1 );
8537- stbir__simdf8_store( decode + 0, of0 );
8538- stbir__simdf8_store( decode + 8, of1 );
8539- #else
8540- stbir__simdi i, o0, o1, o2, o3;
8541- stbir__simdf of0, of1, of2, of3;
8542- STBIR_NO_UNROLL(decode);
8543- stbir__simdi_load( i, input );
8544- stbir__simdi_expand_u8_to_u32( o0,o1,o2,o3,i);
8545- stbir__simdi_convert_i32_to_float( of0, o0 );
8546- stbir__simdi_convert_i32_to_float( of1, o1 );
8547- stbir__simdi_convert_i32_to_float( of2, o2 );
8548- stbir__simdi_convert_i32_to_float( of3, o3 );
8549- stbir__decode_simdf4_flip( of0 );
8550- stbir__decode_simdf4_flip( of1 );
8551- stbir__decode_simdf4_flip( of2 );
8552- stbir__decode_simdf4_flip( of3 );
8553- stbir__simdf_store( decode + 0, of0 );
8554- stbir__simdf_store( decode + 4, of1 );
8555- stbir__simdf_store( decode + 8, of2 );
8556- stbir__simdf_store( decode + 12, of3 );
8557-#endif
8558- decode += 16;
8559- input += 16;
8560- if ( decode <= decode_end )
8561- continue;
8562- if ( decode == ( decode_end + 16 ) )
8563- break;
8564- decode = decode_end; // backup and do last couple
8565- input = end_input_m16;
8566- }
8567- return decode_end + 16;
8568- }
8569- #endif
8570-
8571- // try to do blocks of 4 when you can
8572- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8573- decode += 4;
8574- STBIR_SIMD_NO_UNROLL_LOOP_START
8575- while( decode <= decode_end )
8576- {
8577- STBIR_SIMD_NO_UNROLL(decode);
8578- decode[0-4] = ((float)(input[stbir__decode_order0]));
8579- decode[1-4] = ((float)(input[stbir__decode_order1]));
8580- decode[2-4] = ((float)(input[stbir__decode_order2]));
8581- decode[3-4] = ((float)(input[stbir__decode_order3]));
8582- decode += 4;
8583- input += 4;
8584- }
8585- decode -= 4;
8586- #endif
8587-
8588- // do the remnants
8589- #if stbir__coder_min_num < 4
8590- STBIR_NO_UNROLL_LOOP_START
8591- while( decode < decode_end )
8592- {
8593- STBIR_NO_UNROLL(decode);
8594- decode[0] = ((float)(input[stbir__decode_order0]));
8595- #if stbir__coder_min_num >= 2
8596- decode[1] = ((float)(input[stbir__decode_order1]));
8597- #endif
8598- #if stbir__coder_min_num >= 3
8599- decode[2] = ((float)(input[stbir__decode_order2]));
8600- #endif
8601- decode += stbir__coder_min_num;
8602- input += stbir__coder_min_num;
8603- }
8604- #endif
8605- return decode_end;
8606-}
8607-
8608-static void STBIR__CODER_NAME( stbir__encode_uint8_linear )( void * outputp, int width_times_channels, float const * encode )
8609-{
8610- unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char *) outputp;
8611- unsigned char * end_output = ( (unsigned char *) output ) + width_times_channels;
8612-
8613- #ifdef STBIR_SIMD
8614- if ( width_times_channels >= stbir__simdfX_float_count*2 )
8615- {
8616- float const * end_encode_m8 = encode + width_times_channels - stbir__simdfX_float_count*2;
8617- end_output -= stbir__simdfX_float_count*2;
8618- STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
8619- for(;;)
8620- {
8621- stbir__simdfX e0, e1;
8622- stbir__simdi i;
8623- STBIR_SIMD_NO_UNROLL(encode);
8624- stbir__simdfX_add_mem( e0, STBIR_simd_point5X, encode );
8625- stbir__simdfX_add_mem( e1, STBIR_simd_point5X, encode+stbir__simdfX_float_count );
8626- stbir__encode_simdfX_unflip( e0 );
8627- stbir__encode_simdfX_unflip( e1 );
8628- #ifdef STBIR_SIMD8
8629- stbir__simdf8_pack_to_16bytes( i, e0, e1 );
8630- stbir__simdi_store( output, i );
8631- #else
8632- stbir__simdf_pack_to_8bytes( i, e0, e1 );
8633- stbir__simdi_store2( output, i );
8634- #endif
8635- encode += stbir__simdfX_float_count*2;
8636- output += stbir__simdfX_float_count*2;
8637- if ( output <= end_output )
8638- continue;
8639- if ( output == ( end_output + stbir__simdfX_float_count*2 ) )
8640- break;
8641- output = end_output; // backup and do last couple
8642- encode = end_encode_m8;
8643- }
8644- return;
8645- }
8646-
8647- // try to do blocks of 4 when you can
8648- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8649- output += 4;
8650- STBIR_NO_UNROLL_LOOP_START
8651- while( output <= end_output )
8652- {
8653- stbir__simdf e0;
8654- stbir__simdi i0;
8655- STBIR_NO_UNROLL(encode);
8656- stbir__simdf_load( e0, encode );
8657- stbir__simdf_add( e0, STBIR__CONSTF(STBIR_simd_point5), e0 );
8658- stbir__encode_simdf4_unflip( e0 );
8659- stbir__simdf_pack_to_8bytes( i0, e0, e0 ); // only use first 4
8660- *(int*)(output-4) = stbir__simdi_to_int( i0 );
8661- output += 4;
8662- encode += 4;
8663- }
8664- output -= 4;
8665- #endif
8666-
8667- #else
8668-
8669- // try to do blocks of 4 when you can
8670- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8671- output += 4;
8672- while( output <= end_output )
8673- {
8674- float f;
8675- f = encode[stbir__encode_order0] + 0.5f; STBIR_CLAMP(f, 0, 255); output[0-4] = (unsigned char)f;
8676- f = encode[stbir__encode_order1] + 0.5f; STBIR_CLAMP(f, 0, 255); output[1-4] = (unsigned char)f;
8677- f = encode[stbir__encode_order2] + 0.5f; STBIR_CLAMP(f, 0, 255); output[2-4] = (unsigned char)f;
8678- f = encode[stbir__encode_order3] + 0.5f; STBIR_CLAMP(f, 0, 255); output[3-4] = (unsigned char)f;
8679- output += 4;
8680- encode += 4;
8681- }
8682- output -= 4;
8683- #endif
8684-
8685- #endif
8686-
8687- // do the remnants
8688- #if stbir__coder_min_num < 4
8689- STBIR_NO_UNROLL_LOOP_START
8690- while( output < end_output )
8691- {
8692- float f;
8693- STBIR_NO_UNROLL(encode);
8694- f = encode[stbir__encode_order0] + 0.5f; STBIR_CLAMP(f, 0, 255); output[0] = (unsigned char)f;
8695- #if stbir__coder_min_num >= 2
8696- f = encode[stbir__encode_order1] + 0.5f; STBIR_CLAMP(f, 0, 255); output[1] = (unsigned char)f;
8697- #endif
8698- #if stbir__coder_min_num >= 3
8699- f = encode[stbir__encode_order2] + 0.5f; STBIR_CLAMP(f, 0, 255); output[2] = (unsigned char)f;
8700- #endif
8701- output += stbir__coder_min_num;
8702- encode += stbir__coder_min_num;
8703- }
8704- #endif
8705-}
8706-
8707-static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb)( float * decodep, int width_times_channels, void const * inputp )
8708-{
8709- float STBIR_STREAMOUT_PTR( * ) decode = decodep;
8710- float * decode_end = (float*) decode + width_times_channels;
8711- unsigned char const * input = (unsigned char const *)inputp;
8712-
8713- // try to do blocks of 4 when you can
8714- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8715- decode += 4;
8716- while( decode <= decode_end )
8717- {
8718- decode[0-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order0 ] ];
8719- decode[1-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order1 ] ];
8720- decode[2-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order2 ] ];
8721- decode[3-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order3 ] ];
8722- decode += 4;
8723- input += 4;
8724- }
8725- decode -= 4;
8726- #endif
8727-
8728- // do the remnants
8729- #if stbir__coder_min_num < 4
8730- STBIR_NO_UNROLL_LOOP_START
8731- while( decode < decode_end )
8732- {
8733- STBIR_NO_UNROLL(decode);
8734- decode[0] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order0 ] ];
8735- #if stbir__coder_min_num >= 2
8736- decode[1] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order1 ] ];
8737- #endif
8738- #if stbir__coder_min_num >= 3
8739- decode[2] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order2 ] ];
8740- #endif
8741- decode += stbir__coder_min_num;
8742- input += stbir__coder_min_num;
8743- }
8744- #endif
8745- return decode_end;
8746-}
8747-
8748-#define stbir__min_max_shift20( i, f ) \
8749- stbir__simdf_max( f, f, stbir_simdf_casti(STBIR__CONSTI( STBIR_almost_zero )) ); \
8750- stbir__simdf_min( f, f, stbir_simdf_casti(STBIR__CONSTI( STBIR_almost_one )) ); \
8751- stbir__simdi_32shr( i, stbir_simdi_castf( f ), 20 );
8752-
8753-#define stbir__scale_and_convert( i, f ) \
8754- stbir__simdf_madd( f, STBIR__CONSTF( STBIR_simd_point5 ), STBIR__CONSTF( STBIR_max_uint8_as_float ), f ); \
8755- stbir__simdf_max( f, f, stbir__simdf_zeroP() ); \
8756- stbir__simdf_min( f, f, STBIR__CONSTF( STBIR_max_uint8_as_float ) ); \
8757- stbir__simdf_convert_float_to_i32( i, f );
8758-
8759-#define stbir__linear_to_srgb_finish( i, f ) \
8760-{ \
8761- stbir__simdi temp; \
8762- stbir__simdi_32shr( temp, stbir_simdi_castf( f ), 12 ) ; \
8763- stbir__simdi_and( temp, temp, STBIR__CONSTI(STBIR_mantissa_mask) ); \
8764- stbir__simdi_or( temp, temp, STBIR__CONSTI(STBIR_topscale) ); \
8765- stbir__simdi_16madd( i, i, temp ); \
8766- stbir__simdi_32shr( i, i, 16 ); \
8767-}
8768-
8769-#define stbir__simdi_table_lookup2( v0,v1, table ) \
8770-{ \
8771- stbir__simdi_u32 temp0,temp1; \
8772- temp0.m128i_i128 = v0; \
8773- temp1.m128i_i128 = v1; \
8774- temp0.m128i_u32[0] = table[temp0.m128i_i32[0]]; temp0.m128i_u32[1] = table[temp0.m128i_i32[1]]; temp0.m128i_u32[2] = table[temp0.m128i_i32[2]]; temp0.m128i_u32[3] = table[temp0.m128i_i32[3]]; \
8775- temp1.m128i_u32[0] = table[temp1.m128i_i32[0]]; temp1.m128i_u32[1] = table[temp1.m128i_i32[1]]; temp1.m128i_u32[2] = table[temp1.m128i_i32[2]]; temp1.m128i_u32[3] = table[temp1.m128i_i32[3]]; \
8776- v0 = temp0.m128i_i128; \
8777- v1 = temp1.m128i_i128; \
8778-}
8779-
8780-#define stbir__simdi_table_lookup3( v0,v1,v2, table ) \
8781-{ \
8782- stbir__simdi_u32 temp0,temp1,temp2; \
8783- temp0.m128i_i128 = v0; \
8784- temp1.m128i_i128 = v1; \
8785- temp2.m128i_i128 = v2; \
8786- temp0.m128i_u32[0] = table[temp0.m128i_i32[0]]; temp0.m128i_u32[1] = table[temp0.m128i_i32[1]]; temp0.m128i_u32[2] = table[temp0.m128i_i32[2]]; temp0.m128i_u32[3] = table[temp0.m128i_i32[3]]; \
8787- temp1.m128i_u32[0] = table[temp1.m128i_i32[0]]; temp1.m128i_u32[1] = table[temp1.m128i_i32[1]]; temp1.m128i_u32[2] = table[temp1.m128i_i32[2]]; temp1.m128i_u32[3] = table[temp1.m128i_i32[3]]; \
8788- temp2.m128i_u32[0] = table[temp2.m128i_i32[0]]; temp2.m128i_u32[1] = table[temp2.m128i_i32[1]]; temp2.m128i_u32[2] = table[temp2.m128i_i32[2]]; temp2.m128i_u32[3] = table[temp2.m128i_i32[3]]; \
8789- v0 = temp0.m128i_i128; \
8790- v1 = temp1.m128i_i128; \
8791- v2 = temp2.m128i_i128; \
8792-}
8793-
8794-#define stbir__simdi_table_lookup4( v0,v1,v2,v3, table ) \
8795-{ \
8796- stbir__simdi_u32 temp0,temp1,temp2,temp3; \
8797- temp0.m128i_i128 = v0; \
8798- temp1.m128i_i128 = v1; \
8799- temp2.m128i_i128 = v2; \
8800- temp3.m128i_i128 = v3; \
8801- temp0.m128i_u32[0] = table[temp0.m128i_i32[0]]; temp0.m128i_u32[1] = table[temp0.m128i_i32[1]]; temp0.m128i_u32[2] = table[temp0.m128i_i32[2]]; temp0.m128i_u32[3] = table[temp0.m128i_i32[3]]; \
8802- temp1.m128i_u32[0] = table[temp1.m128i_i32[0]]; temp1.m128i_u32[1] = table[temp1.m128i_i32[1]]; temp1.m128i_u32[2] = table[temp1.m128i_i32[2]]; temp1.m128i_u32[3] = table[temp1.m128i_i32[3]]; \
8803- temp2.m128i_u32[0] = table[temp2.m128i_i32[0]]; temp2.m128i_u32[1] = table[temp2.m128i_i32[1]]; temp2.m128i_u32[2] = table[temp2.m128i_i32[2]]; temp2.m128i_u32[3] = table[temp2.m128i_i32[3]]; \
8804- temp3.m128i_u32[0] = table[temp3.m128i_i32[0]]; temp3.m128i_u32[1] = table[temp3.m128i_i32[1]]; temp3.m128i_u32[2] = table[temp3.m128i_i32[2]]; temp3.m128i_u32[3] = table[temp3.m128i_i32[3]]; \
8805- v0 = temp0.m128i_i128; \
8806- v1 = temp1.m128i_i128; \
8807- v2 = temp2.m128i_i128; \
8808- v3 = temp3.m128i_i128; \
8809-}
8810-
8811-static void STBIR__CODER_NAME( stbir__encode_uint8_srgb )( void * outputp, int width_times_channels, float const * encode )
8812-{
8813- unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
8814- unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
8815-
8816- #ifdef STBIR_SIMD
8817-
8818- if ( width_times_channels >= 16 )
8819- {
8820- float const * end_encode_m16 = encode + width_times_channels - 16;
8821- end_output -= 16;
8822- STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
8823- for(;;)
8824- {
8825- stbir__simdf f0, f1, f2, f3;
8826- stbir__simdi i0, i1, i2, i3;
8827- STBIR_SIMD_NO_UNROLL(encode);
8828-
8829- stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
8830-
8831- stbir__min_max_shift20( i0, f0 );
8832- stbir__min_max_shift20( i1, f1 );
8833- stbir__min_max_shift20( i2, f2 );
8834- stbir__min_max_shift20( i3, f3 );
8835-
8836- stbir__simdi_table_lookup4( i0, i1, i2, i3, ( fp32_to_srgb8_tab4 - (127-13)*8 ) );
8837-
8838- stbir__linear_to_srgb_finish( i0, f0 );
8839- stbir__linear_to_srgb_finish( i1, f1 );
8840- stbir__linear_to_srgb_finish( i2, f2 );
8841- stbir__linear_to_srgb_finish( i3, f3 );
8842-
8843- stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
8844-
8845- encode += 16;
8846- output += 16;
8847- if ( output <= end_output )
8848- continue;
8849- if ( output == ( end_output + 16 ) )
8850- break;
8851- output = end_output; // backup and do last couple
8852- encode = end_encode_m16;
8853- }
8854- return;
8855- }
8856- #endif
8857-
8858- // try to do blocks of 4 when you can
8859- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8860- output += 4;
8861- STBIR_SIMD_NO_UNROLL_LOOP_START
8862- while ( output <= end_output )
8863- {
8864- STBIR_SIMD_NO_UNROLL(encode);
8865-
8866- output[0-4] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order0] );
8867- output[1-4] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order1] );
8868- output[2-4] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order2] );
8869- output[3-4] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order3] );
8870-
8871- output += 4;
8872- encode += 4;
8873- }
8874- output -= 4;
8875- #endif
8876-
8877- // do the remnants
8878- #if stbir__coder_min_num < 4
8879- STBIR_NO_UNROLL_LOOP_START
8880- while( output < end_output )
8881- {
8882- STBIR_NO_UNROLL(encode);
8883- output[0] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order0] );
8884- #if stbir__coder_min_num >= 2
8885- output[1] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order1] );
8886- #endif
8887- #if stbir__coder_min_num >= 3
8888- output[2] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order2] );
8889- #endif
8890- output += stbir__coder_min_num;
8891- encode += stbir__coder_min_num;
8892- }
8893- #endif
8894-}
8895-
8896-#if ( stbir__coder_min_num == 4 ) || ( ( stbir__coder_min_num == 1 ) && ( !defined(stbir__decode_swizzle) ) )
8897-
8898-static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb4_linearalpha)( float * decodep, int width_times_channels, void const * inputp )
8899-{
8900- float STBIR_STREAMOUT_PTR( * ) decode = decodep;
8901- float * decode_end = (float*) decode + width_times_channels;
8902- unsigned char const * input = (unsigned char const *)inputp;
8903-
8904- do {
8905- decode[0] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
8906- decode[1] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order1] ];
8907- decode[2] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order2] ];
8908- decode[3] = ( (float) input[stbir__decode_order3] ) * stbir__max_uint8_as_float_inverted;
8909- input += 4;
8910- decode += 4;
8911- } while( decode < decode_end );
8912- return decode_end;
8913-}
8914-
8915-
8916-static void STBIR__CODER_NAME( stbir__encode_uint8_srgb4_linearalpha )( void * outputp, int width_times_channels, float const * encode )
8917-{
8918- unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
8919- unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
8920-
8921- #ifdef STBIR_SIMD
8922-
8923- if ( width_times_channels >= 16 )
8924- {
8925- float const * end_encode_m16 = encode + width_times_channels - 16;
8926- end_output -= 16;
8927- STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
8928- for(;;)
8929- {
8930- stbir__simdf f0, f1, f2, f3;
8931- stbir__simdi i0, i1, i2, i3;
8932-
8933- STBIR_SIMD_NO_UNROLL(encode);
8934- stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
8935-
8936- stbir__min_max_shift20( i0, f0 );
8937- stbir__min_max_shift20( i1, f1 );
8938- stbir__min_max_shift20( i2, f2 );
8939- stbir__scale_and_convert( i3, f3 );
8940-
8941- stbir__simdi_table_lookup3( i0, i1, i2, ( fp32_to_srgb8_tab4 - (127-13)*8 ) );
8942-
8943- stbir__linear_to_srgb_finish( i0, f0 );
8944- stbir__linear_to_srgb_finish( i1, f1 );
8945- stbir__linear_to_srgb_finish( i2, f2 );
8946-
8947- stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
8948-
8949- output += 16;
8950- encode += 16;
8951-
8952- if ( output <= end_output )
8953- continue;
8954- if ( output == ( end_output + 16 ) )
8955- break;
8956- output = end_output; // backup and do last couple
8957- encode = end_encode_m16;
8958- }
8959- return;
8960- }
8961- #endif
8962-
8963- STBIR_SIMD_NO_UNROLL_LOOP_START
8964- do {
8965- float f;
8966- STBIR_SIMD_NO_UNROLL(encode);
8967-
8968- output[stbir__decode_order0] = stbir__linear_to_srgb_uchar( encode[0] );
8969- output[stbir__decode_order1] = stbir__linear_to_srgb_uchar( encode[1] );
8970- output[stbir__decode_order2] = stbir__linear_to_srgb_uchar( encode[2] );
8971-
8972- f = encode[3] * stbir__max_uint8_as_float + 0.5f;
8973- STBIR_CLAMP(f, 0, 255);
8974- output[stbir__decode_order3] = (unsigned char) f;
8975-
8976- output += 4;
8977- encode += 4;
8978- } while( output < end_output );
8979-}
8980-
8981-#endif
8982-
8983-#if ( stbir__coder_min_num == 2 ) || ( ( stbir__coder_min_num == 1 ) && ( !defined(stbir__decode_swizzle) ) )
8984-
8985-static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb2_linearalpha)( float * decodep, int width_times_channels, void const * inputp )
8986-{
8987- float STBIR_STREAMOUT_PTR( * ) decode = decodep;
8988- float * decode_end = (float*) decode + width_times_channels;
8989- unsigned char const * input = (unsigned char const *)inputp;
8990-
8991- decode += 4;
8992- while( decode <= decode_end )
8993- {
8994- decode[0-4] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
8995- decode[1-4] = ( (float) input[stbir__decode_order1] ) * stbir__max_uint8_as_float_inverted;
8996- decode[2-4] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0+2] ];
8997- decode[3-4] = ( (float) input[stbir__decode_order1+2] ) * stbir__max_uint8_as_float_inverted;
8998- input += 4;
8999- decode += 4;
9000- }
9001- decode -= 4;
9002- if( decode < decode_end )
9003- {
9004- decode[0] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
9005- decode[1] = ( (float) input[stbir__decode_order1] ) * stbir__max_uint8_as_float_inverted;
9006- }
9007- return decode_end;
9008-}
9009-
9010-static void STBIR__CODER_NAME( stbir__encode_uint8_srgb2_linearalpha )( void * outputp, int width_times_channels, float const * encode )
9011-{
9012- unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
9013- unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
9014-
9015- #ifdef STBIR_SIMD
9016-
9017- if ( width_times_channels >= 16 )
9018- {
9019- float const * end_encode_m16 = encode + width_times_channels - 16;
9020- end_output -= 16;
9021- STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
9022- for(;;)
9023- {
9024- stbir__simdf f0, f1, f2, f3;
9025- stbir__simdi i0, i1, i2, i3;
9026-
9027- STBIR_SIMD_NO_UNROLL(encode);
9028- stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
9029-
9030- stbir__min_max_shift20( i0, f0 );
9031- stbir__scale_and_convert( i1, f1 );
9032- stbir__min_max_shift20( i2, f2 );
9033- stbir__scale_and_convert( i3, f3 );
9034-
9035- stbir__simdi_table_lookup2( i0, i2, ( fp32_to_srgb8_tab4 - (127-13)*8 ) );
9036-
9037- stbir__linear_to_srgb_finish( i0, f0 );
9038- stbir__linear_to_srgb_finish( i2, f2 );
9039-
9040- stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
9041-
9042- output += 16;
9043- encode += 16;
9044- if ( output <= end_output )
9045- continue;
9046- if ( output == ( end_output + 16 ) )
9047- break;
9048- output = end_output; // backup and do last couple
9049- encode = end_encode_m16;
9050- }
9051- return;
9052- }
9053- #endif
9054-
9055- STBIR_SIMD_NO_UNROLL_LOOP_START
9056- do {
9057- float f;
9058- STBIR_SIMD_NO_UNROLL(encode);
9059-
9060- output[stbir__decode_order0] = stbir__linear_to_srgb_uchar( encode[0] );
9061-
9062- f = encode[1] * stbir__max_uint8_as_float + 0.5f;
9063- STBIR_CLAMP(f, 0, 255);
9064- output[stbir__decode_order1] = (unsigned char) f;
9065-
9066- output += 2;
9067- encode += 2;
9068- } while( output < end_output );
9069-}
9070-
9071-#endif
9072-
9073-static float * STBIR__CODER_NAME(stbir__decode_uint16_linear_scaled)( float * decodep, int width_times_channels, void const * inputp )
9074-{
9075- float STBIR_STREAMOUT_PTR( * ) decode = decodep;
9076- float * decode_end = (float*) decode + width_times_channels;
9077- unsigned short const * input = (unsigned short const *)inputp;
9078-
9079- #ifdef STBIR_SIMD
9080- unsigned short const * end_input_m8 = input + width_times_channels - 8;
9081- if ( width_times_channels >= 8 )
9082- {
9083- decode_end -= 8;
9084- STBIR_NO_UNROLL_LOOP_START_INF_FOR
9085- for(;;)
9086- {
9087- #ifdef STBIR_SIMD8
9088- stbir__simdi i; stbir__simdi8 o;
9089- stbir__simdf8 of;
9090- STBIR_NO_UNROLL(decode);
9091- stbir__simdi_load( i, input );
9092- stbir__simdi8_expand_u16_to_u32( o, i );
9093- stbir__simdi8_convert_i32_to_float( of, o );
9094- stbir__simdf8_mult( of, of, STBIR_max_uint16_as_float_inverted8);
9095- stbir__decode_simdf8_flip( of );
9096- stbir__simdf8_store( decode + 0, of );
9097- #else
9098- stbir__simdi i, o0, o1;
9099- stbir__simdf of0, of1;
9100- STBIR_NO_UNROLL(decode);
9101- stbir__simdi_load( i, input );
9102- stbir__simdi_expand_u16_to_u32( o0,o1,i );
9103- stbir__simdi_convert_i32_to_float( of0, o0 );
9104- stbir__simdi_convert_i32_to_float( of1, o1 );
9105- stbir__simdf_mult( of0, of0, STBIR__CONSTF(STBIR_max_uint16_as_float_inverted) );
9106- stbir__simdf_mult( of1, of1, STBIR__CONSTF(STBIR_max_uint16_as_float_inverted));
9107- stbir__decode_simdf4_flip( of0 );
9108- stbir__decode_simdf4_flip( of1 );
9109- stbir__simdf_store( decode + 0, of0 );
9110- stbir__simdf_store( decode + 4, of1 );
9111- #endif
9112- decode += 8;
9113- input += 8;
9114- if ( decode <= decode_end )
9115- continue;
9116- if ( decode == ( decode_end + 8 ) )
9117- break;
9118- decode = decode_end; // backup and do last couple
9119- input = end_input_m8;
9120- }
9121- return decode_end + 8;
9122- }
9123- #endif
9124-
9125- // try to do blocks of 4 when you can
9126- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9127- decode += 4;
9128- STBIR_SIMD_NO_UNROLL_LOOP_START
9129- while( decode <= decode_end )
9130- {
9131- STBIR_SIMD_NO_UNROLL(decode);
9132- decode[0-4] = ((float)(input[stbir__decode_order0])) * stbir__max_uint16_as_float_inverted;
9133- decode[1-4] = ((float)(input[stbir__decode_order1])) * stbir__max_uint16_as_float_inverted;
9134- decode[2-4] = ((float)(input[stbir__decode_order2])) * stbir__max_uint16_as_float_inverted;
9135- decode[3-4] = ((float)(input[stbir__decode_order3])) * stbir__max_uint16_as_float_inverted;
9136- decode += 4;
9137- input += 4;
9138- }
9139- decode -= 4;
9140- #endif
9141-
9142- // do the remnants
9143- #if stbir__coder_min_num < 4
9144- STBIR_NO_UNROLL_LOOP_START
9145- while( decode < decode_end )
9146- {
9147- STBIR_NO_UNROLL(decode);
9148- decode[0] = ((float)(input[stbir__decode_order0])) * stbir__max_uint16_as_float_inverted;
9149- #if stbir__coder_min_num >= 2
9150- decode[1] = ((float)(input[stbir__decode_order1])) * stbir__max_uint16_as_float_inverted;
9151- #endif
9152- #if stbir__coder_min_num >= 3
9153- decode[2] = ((float)(input[stbir__decode_order2])) * stbir__max_uint16_as_float_inverted;
9154- #endif
9155- decode += stbir__coder_min_num;
9156- input += stbir__coder_min_num;
9157- }
9158- #endif
9159- return decode_end;
9160-}
9161-
9162-
9163-static void STBIR__CODER_NAME(stbir__encode_uint16_linear_scaled)( void * outputp, int width_times_channels, float const * encode )
9164-{
9165- unsigned short STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned short*) outputp;
9166- unsigned short * end_output = ( (unsigned short*) output ) + width_times_channels;
9167-
9168- #ifdef STBIR_SIMD
9169- {
9170- if ( width_times_channels >= stbir__simdfX_float_count*2 )
9171- {
9172- float const * end_encode_m8 = encode + width_times_channels - stbir__simdfX_float_count*2;
9173- end_output -= stbir__simdfX_float_count*2;
9174- STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
9175- for(;;)
9176- {
9177- stbir__simdfX e0, e1;
9178- stbir__simdiX i;
9179- STBIR_SIMD_NO_UNROLL(encode);
9180- stbir__simdfX_madd_mem( e0, STBIR_simd_point5X, STBIR_max_uint16_as_floatX, encode );
9181- stbir__simdfX_madd_mem( e1, STBIR_simd_point5X, STBIR_max_uint16_as_floatX, encode+stbir__simdfX_float_count );
9182- stbir__encode_simdfX_unflip( e0 );
9183- stbir__encode_simdfX_unflip( e1 );
9184- stbir__simdfX_pack_to_words( i, e0, e1 );
9185- stbir__simdiX_store( output, i );
9186- encode += stbir__simdfX_float_count*2;
9187- output += stbir__simdfX_float_count*2;
9188- if ( output <= end_output )
9189- continue;
9190- if ( output == ( end_output + stbir__simdfX_float_count*2 ) )
9191- break;
9192- output = end_output; // backup and do last couple
9193- encode = end_encode_m8;
9194- }
9195- return;
9196- }
9197- }
9198-
9199- // try to do blocks of 4 when you can
9200- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9201- output += 4;
9202- STBIR_NO_UNROLL_LOOP_START
9203- while( output <= end_output )
9204- {
9205- stbir__simdf e;
9206- stbir__simdi i;
9207- STBIR_NO_UNROLL(encode);
9208- stbir__simdf_load( e, encode );
9209- stbir__simdf_madd( e, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), e );
9210- stbir__encode_simdf4_unflip( e );
9211- stbir__simdf_pack_to_8words( i, e, e ); // only use first 4
9212- stbir__simdi_store2( output-4, i );
9213- output += 4;
9214- encode += 4;
9215- }
9216- output -= 4;
9217- #endif
9218-
9219- // do the remnants
9220- #if stbir__coder_min_num < 4
9221- STBIR_NO_UNROLL_LOOP_START
9222- while( output < end_output )
9223- {
9224- stbir__simdf e;
9225- STBIR_NO_UNROLL(encode);
9226- stbir__simdf_madd1_mem( e, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order0 ); output[0] = stbir__simdf_convert_float_to_short( e );
9227- #if stbir__coder_min_num >= 2
9228- stbir__simdf_madd1_mem( e, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order1 ); output[1] = stbir__simdf_convert_float_to_short( e );
9229- #endif
9230- #if stbir__coder_min_num >= 3
9231- stbir__simdf_madd1_mem( e, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order2 ); output[2] = stbir__simdf_convert_float_to_short( e );
9232- #endif
9233- output += stbir__coder_min_num;
9234- encode += stbir__coder_min_num;
9235- }
9236- #endif
9237-
9238- #else
9239-
9240- // try to do blocks of 4 when you can
9241- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9242- output += 4;
9243- STBIR_SIMD_NO_UNROLL_LOOP_START
9244- while( output <= end_output )
9245- {
9246- float f;
9247- STBIR_SIMD_NO_UNROLL(encode);
9248- f = encode[stbir__encode_order0] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[0-4] = (unsigned short)f;
9249- f = encode[stbir__encode_order1] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[1-4] = (unsigned short)f;
9250- f = encode[stbir__encode_order2] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[2-4] = (unsigned short)f;
9251- f = encode[stbir__encode_order3] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[3-4] = (unsigned short)f;
9252- output += 4;
9253- encode += 4;
9254- }
9255- output -= 4;
9256- #endif
9257-
9258- // do the remnants
9259- #if stbir__coder_min_num < 4
9260- STBIR_NO_UNROLL_LOOP_START
9261- while( output < end_output )
9262- {
9263- float f;
9264- STBIR_NO_UNROLL(encode);
9265- f = encode[stbir__encode_order0] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[0] = (unsigned short)f;
9266- #if stbir__coder_min_num >= 2
9267- f = encode[stbir__encode_order1] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[1] = (unsigned short)f;
9268- #endif
9269- #if stbir__coder_min_num >= 3
9270- f = encode[stbir__encode_order2] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[2] = (unsigned short)f;
9271- #endif
9272- output += stbir__coder_min_num;
9273- encode += stbir__coder_min_num;
9274- }
9275- #endif
9276- #endif
9277-}
9278-
9279-static float * STBIR__CODER_NAME(stbir__decode_uint16_linear)( float * decodep, int width_times_channels, void const * inputp )
9280-{
9281- float STBIR_STREAMOUT_PTR( * ) decode = decodep;
9282- float * decode_end = (float*) decode + width_times_channels;
9283- unsigned short const * input = (unsigned short const *)inputp;
9284-
9285- #ifdef STBIR_SIMD
9286- unsigned short const * end_input_m8 = input + width_times_channels - 8;
9287- if ( width_times_channels >= 8 )
9288- {
9289- decode_end -= 8;
9290- STBIR_NO_UNROLL_LOOP_START_INF_FOR
9291- for(;;)
9292- {
9293- #ifdef STBIR_SIMD8
9294- stbir__simdi i; stbir__simdi8 o;
9295- stbir__simdf8 of;
9296- STBIR_NO_UNROLL(decode);
9297- stbir__simdi_load( i, input );
9298- stbir__simdi8_expand_u16_to_u32( o, i );
9299- stbir__simdi8_convert_i32_to_float( of, o );
9300- stbir__decode_simdf8_flip( of );
9301- stbir__simdf8_store( decode + 0, of );
9302- #else
9303- stbir__simdi i, o0, o1;
9304- stbir__simdf of0, of1;
9305- STBIR_NO_UNROLL(decode);
9306- stbir__simdi_load( i, input );
9307- stbir__simdi_expand_u16_to_u32( o0, o1, i );
9308- stbir__simdi_convert_i32_to_float( of0, o0 );
9309- stbir__simdi_convert_i32_to_float( of1, o1 );
9310- stbir__decode_simdf4_flip( of0 );
9311- stbir__decode_simdf4_flip( of1 );
9312- stbir__simdf_store( decode + 0, of0 );
9313- stbir__simdf_store( decode + 4, of1 );
9314- #endif
9315- decode += 8;
9316- input += 8;
9317- if ( decode <= decode_end )
9318- continue;
9319- if ( decode == ( decode_end + 8 ) )
9320- break;
9321- decode = decode_end; // backup and do last couple
9322- input = end_input_m8;
9323- }
9324- return decode_end + 8;
9325- }
9326- #endif
9327-
9328- // try to do blocks of 4 when you can
9329- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9330- decode += 4;
9331- STBIR_SIMD_NO_UNROLL_LOOP_START
9332- while( decode <= decode_end )
9333- {
9334- STBIR_SIMD_NO_UNROLL(decode);
9335- decode[0-4] = ((float)(input[stbir__decode_order0]));
9336- decode[1-4] = ((float)(input[stbir__decode_order1]));
9337- decode[2-4] = ((float)(input[stbir__decode_order2]));
9338- decode[3-4] = ((float)(input[stbir__decode_order3]));
9339- decode += 4;
9340- input += 4;
9341- }
9342- decode -= 4;
9343- #endif
9344-
9345- // do the remnants
9346- #if stbir__coder_min_num < 4
9347- STBIR_NO_UNROLL_LOOP_START
9348- while( decode < decode_end )
9349- {
9350- STBIR_NO_UNROLL(decode);
9351- decode[0] = ((float)(input[stbir__decode_order0]));
9352- #if stbir__coder_min_num >= 2
9353- decode[1] = ((float)(input[stbir__decode_order1]));
9354- #endif
9355- #if stbir__coder_min_num >= 3
9356- decode[2] = ((float)(input[stbir__decode_order2]));
9357- #endif
9358- decode += stbir__coder_min_num;
9359- input += stbir__coder_min_num;
9360- }
9361- #endif
9362- return decode_end;
9363-}
9364-
9365-static void STBIR__CODER_NAME(stbir__encode_uint16_linear)( void * outputp, int width_times_channels, float const * encode )
9366-{
9367- unsigned short STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned short*) outputp;
9368- unsigned short * end_output = ( (unsigned short*) output ) + width_times_channels;
9369-
9370- #ifdef STBIR_SIMD
9371- {
9372- if ( width_times_channels >= stbir__simdfX_float_count*2 )
9373- {
9374- float const * end_encode_m8 = encode + width_times_channels - stbir__simdfX_float_count*2;
9375- end_output -= stbir__simdfX_float_count*2;
9376- STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
9377- for(;;)
9378- {
9379- stbir__simdfX e0, e1;
9380- stbir__simdiX i;
9381- STBIR_SIMD_NO_UNROLL(encode);
9382- stbir__simdfX_add_mem( e0, STBIR_simd_point5X, encode );
9383- stbir__simdfX_add_mem( e1, STBIR_simd_point5X, encode+stbir__simdfX_float_count );
9384- stbir__encode_simdfX_unflip( e0 );
9385- stbir__encode_simdfX_unflip( e1 );
9386- stbir__simdfX_pack_to_words( i, e0, e1 );
9387- stbir__simdiX_store( output, i );
9388- encode += stbir__simdfX_float_count*2;
9389- output += stbir__simdfX_float_count*2;
9390- if ( output <= end_output )
9391- continue;
9392- if ( output == ( end_output + stbir__simdfX_float_count*2 ) )
9393- break;
9394- output = end_output; // backup and do last couple
9395- encode = end_encode_m8;
9396- }
9397- return;
9398- }
9399- }
9400-
9401- // try to do blocks of 4 when you can
9402- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9403- output += 4;
9404- STBIR_NO_UNROLL_LOOP_START
9405- while( output <= end_output )
9406- {
9407- stbir__simdf e;
9408- stbir__simdi i;
9409- STBIR_NO_UNROLL(encode);
9410- stbir__simdf_load( e, encode );
9411- stbir__simdf_add( e, STBIR__CONSTF(STBIR_simd_point5), e );
9412- stbir__encode_simdf4_unflip( e );
9413- stbir__simdf_pack_to_8words( i, e, e ); // only use first 4
9414- stbir__simdi_store2( output-4, i );
9415- output += 4;
9416- encode += 4;
9417- }
9418- output -= 4;
9419- #endif
9420-
9421- #else
9422-
9423- // try to do blocks of 4 when you can
9424- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9425- output += 4;
9426- STBIR_SIMD_NO_UNROLL_LOOP_START
9427- while( output <= end_output )
9428- {
9429- float f;
9430- STBIR_SIMD_NO_UNROLL(encode);
9431- f = encode[stbir__encode_order0] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[0-4] = (unsigned short)f;
9432- f = encode[stbir__encode_order1] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[1-4] = (unsigned short)f;
9433- f = encode[stbir__encode_order2] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[2-4] = (unsigned short)f;
9434- f = encode[stbir__encode_order3] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[3-4] = (unsigned short)f;
9435- output += 4;
9436- encode += 4;
9437- }
9438- output -= 4;
9439- #endif
9440-
9441- #endif
9442-
9443- // do the remnants
9444- #if stbir__coder_min_num < 4
9445- STBIR_NO_UNROLL_LOOP_START
9446- while( output < end_output )
9447- {
9448- float f;
9449- STBIR_NO_UNROLL(encode);
9450- f = encode[stbir__encode_order0] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[0] = (unsigned short)f;
9451- #if stbir__coder_min_num >= 2
9452- f = encode[stbir__encode_order1] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[1] = (unsigned short)f;
9453- #endif
9454- #if stbir__coder_min_num >= 3
9455- f = encode[stbir__encode_order2] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[2] = (unsigned short)f;
9456- #endif
9457- output += stbir__coder_min_num;
9458- encode += stbir__coder_min_num;
9459- }
9460- #endif
9461-}
9462-
9463-static float * STBIR__CODER_NAME(stbir__decode_half_float_linear)( float * decodep, int width_times_channels, void const * inputp )
9464-{
9465- float STBIR_STREAMOUT_PTR( * ) decode = decodep;
9466- float * decode_end = (float*) decode + width_times_channels;
9467- stbir__FP16 const * input = (stbir__FP16 const *)inputp;
9468-
9469- #ifdef STBIR_SIMD
9470- if ( width_times_channels >= 8 )
9471- {
9472- stbir__FP16 const * end_input_m8 = input + width_times_channels - 8;
9473- decode_end -= 8;
9474- STBIR_NO_UNROLL_LOOP_START_INF_FOR
9475- for(;;)
9476- {
9477- STBIR_NO_UNROLL(decode);
9478-
9479- stbir__half_to_float_SIMD( decode, input );
9480- #ifdef stbir__decode_swizzle
9481- #ifdef STBIR_SIMD8
9482- {
9483- stbir__simdf8 of;
9484- stbir__simdf8_load( of, decode );
9485- stbir__decode_simdf8_flip( of );
9486- stbir__simdf8_store( decode, of );
9487- }
9488- #else
9489- {
9490- stbir__simdf of0,of1;
9491- stbir__simdf_load( of0, decode );
9492- stbir__simdf_load( of1, decode+4 );
9493- stbir__decode_simdf4_flip( of0 );
9494- stbir__decode_simdf4_flip( of1 );
9495- stbir__simdf_store( decode, of0 );
9496- stbir__simdf_store( decode+4, of1 );
9497- }
9498- #endif
9499- #endif
9500- decode += 8;
9501- input += 8;
9502- if ( decode <= decode_end )
9503- continue;
9504- if ( decode == ( decode_end + 8 ) )
9505- break;
9506- decode = decode_end; // backup and do last couple
9507- input = end_input_m8;
9508- }
9509- return decode_end + 8;
9510- }
9511- #endif
9512-
9513- // try to do blocks of 4 when you can
9514- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9515- decode += 4;
9516- STBIR_SIMD_NO_UNROLL_LOOP_START
9517- while( decode <= decode_end )
9518- {
9519- STBIR_SIMD_NO_UNROLL(decode);
9520- decode[0-4] = stbir__half_to_float(input[stbir__decode_order0]);
9521- decode[1-4] = stbir__half_to_float(input[stbir__decode_order1]);
9522- decode[2-4] = stbir__half_to_float(input[stbir__decode_order2]);
9523- decode[3-4] = stbir__half_to_float(input[stbir__decode_order3]);
9524- decode += 4;
9525- input += 4;
9526- }
9527- decode -= 4;
9528- #endif
9529-
9530- // do the remnants
9531- #if stbir__coder_min_num < 4
9532- STBIR_NO_UNROLL_LOOP_START
9533- while( decode < decode_end )
9534- {
9535- STBIR_NO_UNROLL(decode);
9536- decode[0] = stbir__half_to_float(input[stbir__decode_order0]);
9537- #if stbir__coder_min_num >= 2
9538- decode[1] = stbir__half_to_float(input[stbir__decode_order1]);
9539- #endif
9540- #if stbir__coder_min_num >= 3
9541- decode[2] = stbir__half_to_float(input[stbir__decode_order2]);
9542- #endif
9543- decode += stbir__coder_min_num;
9544- input += stbir__coder_min_num;
9545- }
9546- #endif
9547- return decode_end;
9548-}
9549-
9550-static void STBIR__CODER_NAME( stbir__encode_half_float_linear )( void * outputp, int width_times_channels, float const * encode )
9551-{
9552- stbir__FP16 STBIR_SIMD_STREAMOUT_PTR( * ) output = (stbir__FP16*) outputp;
9553- stbir__FP16 * end_output = ( (stbir__FP16*) output ) + width_times_channels;
9554-
9555- #ifdef STBIR_SIMD
9556- if ( width_times_channels >= 8 )
9557- {
9558- float const * end_encode_m8 = encode + width_times_channels - 8;
9559- end_output -= 8;
9560- STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
9561- for(;;)
9562- {
9563- STBIR_SIMD_NO_UNROLL(encode);
9564- #ifdef stbir__decode_swizzle
9565- #ifdef STBIR_SIMD8
9566- {
9567- stbir__simdf8 of;
9568- stbir__simdf8_load( of, encode );
9569- stbir__encode_simdf8_unflip( of );
9570- stbir__float_to_half_SIMD( output, (float*)&of );
9571- }
9572- #else
9573- {
9574- stbir__simdf of[2];
9575- stbir__simdf_load( of[0], encode );
9576- stbir__simdf_load( of[1], encode+4 );
9577- stbir__encode_simdf4_unflip( of[0] );
9578- stbir__encode_simdf4_unflip( of[1] );
9579- stbir__float_to_half_SIMD( output, (float*)of );
9580- }
9581- #endif
9582- #else
9583- stbir__float_to_half_SIMD( output, encode );
9584- #endif
9585- encode += 8;
9586- output += 8;
9587- if ( output <= end_output )
9588- continue;
9589- if ( output == ( end_output + 8 ) )
9590- break;
9591- output = end_output; // backup and do last couple
9592- encode = end_encode_m8;
9593- }
9594- return;
9595- }
9596- #endif
9597-
9598- // try to do blocks of 4 when you can
9599- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9600- output += 4;
9601- STBIR_SIMD_NO_UNROLL_LOOP_START
9602- while( output <= end_output )
9603- {
9604- STBIR_SIMD_NO_UNROLL(output);
9605- output[0-4] = stbir__float_to_half(encode[stbir__encode_order0]);
9606- output[1-4] = stbir__float_to_half(encode[stbir__encode_order1]);
9607- output[2-4] = stbir__float_to_half(encode[stbir__encode_order2]);
9608- output[3-4] = stbir__float_to_half(encode[stbir__encode_order3]);
9609- output += 4;
9610- encode += 4;
9611- }
9612- output -= 4;
9613- #endif
9614-
9615- // do the remnants
9616- #if stbir__coder_min_num < 4
9617- STBIR_NO_UNROLL_LOOP_START
9618- while( output < end_output )
9619- {
9620- STBIR_NO_UNROLL(output);
9621- output[0] = stbir__float_to_half(encode[stbir__encode_order0]);
9622- #if stbir__coder_min_num >= 2
9623- output[1] = stbir__float_to_half(encode[stbir__encode_order1]);
9624- #endif
9625- #if stbir__coder_min_num >= 3
9626- output[2] = stbir__float_to_half(encode[stbir__encode_order2]);
9627- #endif
9628- output += stbir__coder_min_num;
9629- encode += stbir__coder_min_num;
9630- }
9631- #endif
9632-}
9633-
9634-static float * STBIR__CODER_NAME(stbir__decode_float_linear)( float * decodep, int width_times_channels, void const * inputp )
9635-{
9636- #ifdef stbir__decode_swizzle
9637- float STBIR_STREAMOUT_PTR( * ) decode = decodep;
9638- float * decode_end = (float*) decode + width_times_channels;
9639- float const * input = (float const *)inputp;
9640-
9641- #ifdef STBIR_SIMD
9642- if ( width_times_channels >= 16 )
9643- {
9644- float const * end_input_m16 = input + width_times_channels - 16;
9645- decode_end -= 16;
9646- STBIR_NO_UNROLL_LOOP_START_INF_FOR
9647- for(;;)
9648- {
9649- STBIR_NO_UNROLL(decode);
9650- #ifdef stbir__decode_swizzle
9651- #ifdef STBIR_SIMD8
9652- {
9653- stbir__simdf8 of0,of1;
9654- stbir__simdf8_load( of0, input );
9655- stbir__simdf8_load( of1, input+8 );
9656- stbir__decode_simdf8_flip( of0 );
9657- stbir__decode_simdf8_flip( of1 );
9658- stbir__simdf8_store( decode, of0 );
9659- stbir__simdf8_store( decode+8, of1 );
9660- }
9661- #else
9662- {
9663- stbir__simdf of0,of1,of2,of3;
9664- stbir__simdf_load( of0, input );
9665- stbir__simdf_load( of1, input+4 );
9666- stbir__simdf_load( of2, input+8 );
9667- stbir__simdf_load( of3, input+12 );
9668- stbir__decode_simdf4_flip( of0 );
9669- stbir__decode_simdf4_flip( of1 );
9670- stbir__decode_simdf4_flip( of2 );
9671- stbir__decode_simdf4_flip( of3 );
9672- stbir__simdf_store( decode, of0 );
9673- stbir__simdf_store( decode+4, of1 );
9674- stbir__simdf_store( decode+8, of2 );
9675- stbir__simdf_store( decode+12, of3 );
9676- }
9677- #endif
9678- #endif
9679- decode += 16;
9680- input += 16;
9681- if ( decode <= decode_end )
9682- continue;
9683- if ( decode == ( decode_end + 16 ) )
9684- break;
9685- decode = decode_end; // backup and do last couple
9686- input = end_input_m16;
9687- }
9688- return decode_end + 16;
9689- }
9690- #endif
9691-
9692- // try to do blocks of 4 when you can
9693- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9694- decode += 4;
9695- STBIR_SIMD_NO_UNROLL_LOOP_START
9696- while( decode <= decode_end )
9697- {
9698- STBIR_SIMD_NO_UNROLL(decode);
9699- decode[0-4] = input[stbir__decode_order0];
9700- decode[1-4] = input[stbir__decode_order1];
9701- decode[2-4] = input[stbir__decode_order2];
9702- decode[3-4] = input[stbir__decode_order3];
9703- decode += 4;
9704- input += 4;
9705- }
9706- decode -= 4;
9707- #endif
9708-
9709- // do the remnants
9710- #if stbir__coder_min_num < 4
9711- STBIR_NO_UNROLL_LOOP_START
9712- while( decode < decode_end )
9713- {
9714- STBIR_NO_UNROLL(decode);
9715- decode[0] = input[stbir__decode_order0];
9716- #if stbir__coder_min_num >= 2
9717- decode[1] = input[stbir__decode_order1];
9718- #endif
9719- #if stbir__coder_min_num >= 3
9720- decode[2] = input[stbir__decode_order2];
9721- #endif
9722- decode += stbir__coder_min_num;
9723- input += stbir__coder_min_num;
9724- }
9725- #endif
9726- return decode_end;
9727-
9728- #else
9729-
9730- if ( (void*)decodep != inputp )
9731- STBIR_MEMCPY( decodep, inputp, width_times_channels * sizeof( float ) );
9732-
9733- return decodep + width_times_channels;
9734-
9735- #endif
9736-}
9737-
9738-static void STBIR__CODER_NAME( stbir__encode_float_linear )( void * outputp, int width_times_channels, float const * encode )
9739-{
9740- #if !defined( STBIR_FLOAT_HIGH_CLAMP ) && !defined(STBIR_FLOAT_LOW_CLAMP) && !defined(stbir__decode_swizzle)
9741-
9742- if ( (void*)outputp != (void*) encode )
9743- STBIR_MEMCPY( outputp, encode, width_times_channels * sizeof( float ) );
9744-
9745- #else
9746-
9747- float STBIR_SIMD_STREAMOUT_PTR( * ) output = (float*) outputp;
9748- float * end_output = ( (float*) output ) + width_times_channels;
9749-
9750- #ifdef STBIR_FLOAT_HIGH_CLAMP
9751- #define stbir_scalar_hi_clamp( v ) if ( v > STBIR_FLOAT_HIGH_CLAMP ) v = STBIR_FLOAT_HIGH_CLAMP;
9752- #else
9753- #define stbir_scalar_hi_clamp( v )
9754- #endif
9755- #ifdef STBIR_FLOAT_LOW_CLAMP
9756- #define stbir_scalar_lo_clamp( v ) if ( v < STBIR_FLOAT_LOW_CLAMP ) v = STBIR_FLOAT_LOW_CLAMP;
9757- #else
9758- #define stbir_scalar_lo_clamp( v )
9759- #endif
9760-
9761- #ifdef STBIR_SIMD
9762-
9763- #ifdef STBIR_FLOAT_HIGH_CLAMP
9764- const stbir__simdfX high_clamp = stbir__simdf_frepX(STBIR_FLOAT_HIGH_CLAMP);
9765- #endif
9766- #ifdef STBIR_FLOAT_LOW_CLAMP
9767- const stbir__simdfX low_clamp = stbir__simdf_frepX(STBIR_FLOAT_LOW_CLAMP);
9768- #endif
9769-
9770- if ( width_times_channels >= ( stbir__simdfX_float_count * 2 ) )
9771- {
9772- float const * end_encode_m8 = encode + width_times_channels - ( stbir__simdfX_float_count * 2 );
9773- end_output -= ( stbir__simdfX_float_count * 2 );
9774- STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
9775- for(;;)
9776- {
9777- stbir__simdfX e0, e1;
9778- STBIR_SIMD_NO_UNROLL(encode);
9779- stbir__simdfX_load( e0, encode );
9780- stbir__simdfX_load( e1, encode+stbir__simdfX_float_count );
9781-#ifdef STBIR_FLOAT_HIGH_CLAMP
9782- stbir__simdfX_min( e0, e0, high_clamp );
9783- stbir__simdfX_min( e1, e1, high_clamp );
9784-#endif
9785-#ifdef STBIR_FLOAT_LOW_CLAMP
9786- stbir__simdfX_max( e0, e0, low_clamp );
9787- stbir__simdfX_max( e1, e1, low_clamp );
9788-#endif
9789- stbir__encode_simdfX_unflip( e0 );
9790- stbir__encode_simdfX_unflip( e1 );
9791- stbir__simdfX_store( output, e0 );
9792- stbir__simdfX_store( output+stbir__simdfX_float_count, e1 );
9793- encode += stbir__simdfX_float_count * 2;
9794- output += stbir__simdfX_float_count * 2;
9795- if ( output <= end_output )
9796- continue;
9797- if ( output == ( end_output + ( stbir__simdfX_float_count * 2 ) ) )
9798- break;
9799- output = end_output; // backup and do last couple
9800- encode = end_encode_m8;
9801- }
9802- return;
9803- }
9804-
9805- // try to do blocks of 4 when you can
9806- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9807- output += 4;
9808- STBIR_NO_UNROLL_LOOP_START
9809- while( output <= end_output )
9810- {
9811- stbir__simdf e0;
9812- STBIR_NO_UNROLL(encode);
9813- stbir__simdf_load( e0, encode );
9814-#ifdef STBIR_FLOAT_HIGH_CLAMP
9815- stbir__simdf_min( e0, e0, high_clamp );
9816-#endif
9817-#ifdef STBIR_FLOAT_LOW_CLAMP
9818- stbir__simdf_max( e0, e0, low_clamp );
9819-#endif
9820- stbir__encode_simdf4_unflip( e0 );
9821- stbir__simdf_store( output-4, e0 );
9822- output += 4;
9823- encode += 4;
9824- }
9825- output -= 4;
9826- #endif
9827-
9828- #else
9829-
9830- // try to do blocks of 4 when you can
9831- #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9832- output += 4;
9833- STBIR_SIMD_NO_UNROLL_LOOP_START
9834- while( output <= end_output )
9835- {
9836- float e;
9837- STBIR_SIMD_NO_UNROLL(encode);
9838- e = encode[ stbir__encode_order0 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[0-4] = e;
9839- e = encode[ stbir__encode_order1 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[1-4] = e;
9840- e = encode[ stbir__encode_order2 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[2-4] = e;
9841- e = encode[ stbir__encode_order3 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[3-4] = e;
9842- output += 4;
9843- encode += 4;
9844- }
9845- output -= 4;
9846-
9847- #endif
9848-
9849- #endif
9850-
9851- // do the remnants
9852- #if stbir__coder_min_num < 4
9853- STBIR_NO_UNROLL_LOOP_START
9854- while( output < end_output )
9855- {
9856- float e;
9857- STBIR_NO_UNROLL(encode);
9858- e = encode[ stbir__encode_order0 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[0] = e;
9859- #if stbir__coder_min_num >= 2
9860- e = encode[ stbir__encode_order1 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[1] = e;
9861- #endif
9862- #if stbir__coder_min_num >= 3
9863- e = encode[ stbir__encode_order2 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[2] = e;
9864- #endif
9865- output += stbir__coder_min_num;
9866- encode += stbir__coder_min_num;
9867- }
9868- #endif
9869-
9870- #endif
9871-}
9872-
9873-#undef stbir__decode_suffix
9874-#undef stbir__decode_simdf8_flip
9875-#undef stbir__decode_simdf4_flip
9876-#undef stbir__decode_order0
9877-#undef stbir__decode_order1
9878-#undef stbir__decode_order2
9879-#undef stbir__decode_order3
9880-#undef stbir__encode_order0
9881-#undef stbir__encode_order1
9882-#undef stbir__encode_order2
9883-#undef stbir__encode_order3
9884-#undef stbir__encode_simdf8_unflip
9885-#undef stbir__encode_simdf4_unflip
9886-#undef stbir__encode_simdfX_unflip
9887-#undef STBIR__CODER_NAME
9888-#undef stbir__coder_min_num
9889-#undef stbir__decode_swizzle
9890-#undef stbir_scalar_hi_clamp
9891-#undef stbir_scalar_lo_clamp
9892-#undef STB_IMAGE_RESIZE_DO_CODERS
9893-
9894-#elif defined( STB_IMAGE_RESIZE_DO_VERTICALS)
9895-
9896-#ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
9897-#define STBIR_chans( start, end ) STBIR_strs_join14(start,STBIR__vertical_channels,end,_cont)
9898-#else
9899-#define STBIR_chans( start, end ) STBIR_strs_join1(start,STBIR__vertical_channels,end)
9900-#endif
9901-
9902-#if STBIR__vertical_channels >= 1
9903-#define stbIF0( code ) code
9904-#else
9905-#define stbIF0( code )
9906-#endif
9907-#if STBIR__vertical_channels >= 2
9908-#define stbIF1( code ) code
9909-#else
9910-#define stbIF1( code )
9911-#endif
9912-#if STBIR__vertical_channels >= 3
9913-#define stbIF2( code ) code
9914-#else
9915-#define stbIF2( code )
9916-#endif
9917-#if STBIR__vertical_channels >= 4
9918-#define stbIF3( code ) code
9919-#else
9920-#define stbIF3( code )
9921-#endif
9922-#if STBIR__vertical_channels >= 5
9923-#define stbIF4( code ) code
9924-#else
9925-#define stbIF4( code )
9926-#endif
9927-#if STBIR__vertical_channels >= 6
9928-#define stbIF5( code ) code
9929-#else
9930-#define stbIF5( code )
9931-#endif
9932-#if STBIR__vertical_channels >= 7
9933-#define stbIF6( code ) code
9934-#else
9935-#define stbIF6( code )
9936-#endif
9937-#if STBIR__vertical_channels >= 8
9938-#define stbIF7( code ) code
9939-#else
9940-#define stbIF7( code )
9941-#endif
9942-
9943-static void STBIR_chans( stbir__vertical_scatter_with_,_coeffs)( float ** outputs, float const * vertical_coefficients, float const * input, float const * input_end )
9944-{
9945- stbIF0( float STBIR_SIMD_STREAMOUT_PTR( * ) output0 = outputs[0]; float c0s = vertical_coefficients[0]; )
9946- stbIF1( float STBIR_SIMD_STREAMOUT_PTR( * ) output1 = outputs[1]; float c1s = vertical_coefficients[1]; )
9947- stbIF2( float STBIR_SIMD_STREAMOUT_PTR( * ) output2 = outputs[2]; float c2s = vertical_coefficients[2]; )
9948- stbIF3( float STBIR_SIMD_STREAMOUT_PTR( * ) output3 = outputs[3]; float c3s = vertical_coefficients[3]; )
9949- stbIF4( float STBIR_SIMD_STREAMOUT_PTR( * ) output4 = outputs[4]; float c4s = vertical_coefficients[4]; )
9950- stbIF5( float STBIR_SIMD_STREAMOUT_PTR( * ) output5 = outputs[5]; float c5s = vertical_coefficients[5]; )
9951- stbIF6( float STBIR_SIMD_STREAMOUT_PTR( * ) output6 = outputs[6]; float c6s = vertical_coefficients[6]; )
9952- stbIF7( float STBIR_SIMD_STREAMOUT_PTR( * ) output7 = outputs[7]; float c7s = vertical_coefficients[7]; )
9953-
9954- #ifdef STBIR_SIMD
9955- {
9956- stbIF0(stbir__simdfX c0 = stbir__simdf_frepX( c0s ); )
9957- stbIF1(stbir__simdfX c1 = stbir__simdf_frepX( c1s ); )
9958- stbIF2(stbir__simdfX c2 = stbir__simdf_frepX( c2s ); )
9959- stbIF3(stbir__simdfX c3 = stbir__simdf_frepX( c3s ); )
9960- stbIF4(stbir__simdfX c4 = stbir__simdf_frepX( c4s ); )
9961- stbIF5(stbir__simdfX c5 = stbir__simdf_frepX( c5s ); )
9962- stbIF6(stbir__simdfX c6 = stbir__simdf_frepX( c6s ); )
9963- stbIF7(stbir__simdfX c7 = stbir__simdf_frepX( c7s ); )
9964- STBIR_SIMD_NO_UNROLL_LOOP_START
9965- while ( ( (char*)input_end - (char*) input ) >= (16*stbir__simdfX_float_count) )
9966- {
9967- stbir__simdfX o0, o1, o2, o3, r0, r1, r2, r3;
9968- STBIR_SIMD_NO_UNROLL(output0);
9969-
9970- stbir__simdfX_load( r0, input ); stbir__simdfX_load( r1, input+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input+(3*stbir__simdfX_float_count) );
9971-
9972- #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
9973- stbIF0( stbir__simdfX_load( o0, output0 ); stbir__simdfX_load( o1, output0+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output0+(3*stbir__simdfX_float_count) );
9974- stbir__simdfX_madd( o0, o0, r0, c0 ); stbir__simdfX_madd( o1, o1, r1, c0 ); stbir__simdfX_madd( o2, o2, r2, c0 ); stbir__simdfX_madd( o3, o3, r3, c0 );
9975- stbir__simdfX_store( output0, o0 ); stbir__simdfX_store( output0+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output0+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output0+(3*stbir__simdfX_float_count), o3 ); )
9976- stbIF1( stbir__simdfX_load( o0, output1 ); stbir__simdfX_load( o1, output1+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output1+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output1+(3*stbir__simdfX_float_count) );
9977- stbir__simdfX_madd( o0, o0, r0, c1 ); stbir__simdfX_madd( o1, o1, r1, c1 ); stbir__simdfX_madd( o2, o2, r2, c1 ); stbir__simdfX_madd( o3, o3, r3, c1 );
9978- stbir__simdfX_store( output1, o0 ); stbir__simdfX_store( output1+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output1+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output1+(3*stbir__simdfX_float_count), o3 ); )
9979- stbIF2( stbir__simdfX_load( o0, output2 ); stbir__simdfX_load( o1, output2+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output2+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output2+(3*stbir__simdfX_float_count) );
9980- stbir__simdfX_madd( o0, o0, r0, c2 ); stbir__simdfX_madd( o1, o1, r1, c2 ); stbir__simdfX_madd( o2, o2, r2, c2 ); stbir__simdfX_madd( o3, o3, r3, c2 );
9981- stbir__simdfX_store( output2, o0 ); stbir__simdfX_store( output2+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output2+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output2+(3*stbir__simdfX_float_count), o3 ); )
9982- stbIF3( stbir__simdfX_load( o0, output3 ); stbir__simdfX_load( o1, output3+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output3+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output3+(3*stbir__simdfX_float_count) );
9983- stbir__simdfX_madd( o0, o0, r0, c3 ); stbir__simdfX_madd( o1, o1, r1, c3 ); stbir__simdfX_madd( o2, o2, r2, c3 ); stbir__simdfX_madd( o3, o3, r3, c3 );
9984- stbir__simdfX_store( output3, o0 ); stbir__simdfX_store( output3+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output3+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output3+(3*stbir__simdfX_float_count), o3 ); )
9985- stbIF4( stbir__simdfX_load( o0, output4 ); stbir__simdfX_load( o1, output4+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output4+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output4+(3*stbir__simdfX_float_count) );
9986- stbir__simdfX_madd( o0, o0, r0, c4 ); stbir__simdfX_madd( o1, o1, r1, c4 ); stbir__simdfX_madd( o2, o2, r2, c4 ); stbir__simdfX_madd( o3, o3, r3, c4 );
9987- stbir__simdfX_store( output4, o0 ); stbir__simdfX_store( output4+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output4+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output4+(3*stbir__simdfX_float_count), o3 ); )
9988- stbIF5( stbir__simdfX_load( o0, output5 ); stbir__simdfX_load( o1, output5+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output5+(2*stbir__simdfX_float_count)); stbir__simdfX_load( o3, output5+(3*stbir__simdfX_float_count) );
9989- stbir__simdfX_madd( o0, o0, r0, c5 ); stbir__simdfX_madd( o1, o1, r1, c5 ); stbir__simdfX_madd( o2, o2, r2, c5 ); stbir__simdfX_madd( o3, o3, r3, c5 );
9990- stbir__simdfX_store( output5, o0 ); stbir__simdfX_store( output5+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output5+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output5+(3*stbir__simdfX_float_count), o3 ); )
9991- stbIF6( stbir__simdfX_load( o0, output6 ); stbir__simdfX_load( o1, output6+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output6+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output6+(3*stbir__simdfX_float_count) );
9992- stbir__simdfX_madd( o0, o0, r0, c6 ); stbir__simdfX_madd( o1, o1, r1, c6 ); stbir__simdfX_madd( o2, o2, r2, c6 ); stbir__simdfX_madd( o3, o3, r3, c6 );
9993- stbir__simdfX_store( output6, o0 ); stbir__simdfX_store( output6+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output6+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output6+(3*stbir__simdfX_float_count), o3 ); )
9994- stbIF7( stbir__simdfX_load( o0, output7 ); stbir__simdfX_load( o1, output7+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output7+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output7+(3*stbir__simdfX_float_count) );
9995- stbir__simdfX_madd( o0, o0, r0, c7 ); stbir__simdfX_madd( o1, o1, r1, c7 ); stbir__simdfX_madd( o2, o2, r2, c7 ); stbir__simdfX_madd( o3, o3, r3, c7 );
9996- stbir__simdfX_store( output7, o0 ); stbir__simdfX_store( output7+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output7+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output7+(3*stbir__simdfX_float_count), o3 ); )
9997- #else
9998- stbIF0( stbir__simdfX_mult( o0, r0, c0 ); stbir__simdfX_mult( o1, r1, c0 ); stbir__simdfX_mult( o2, r2, c0 ); stbir__simdfX_mult( o3, r3, c0 );
9999- stbir__simdfX_store( output0, o0 ); stbir__simdfX_store( output0+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output0+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output0+(3*stbir__simdfX_float_count), o3 ); )
10000- stbIF1( stbir__simdfX_mult( o0, r0, c1 ); stbir__simdfX_mult( o1, r1, c1 ); stbir__simdfX_mult( o2, r2, c1 ); stbir__simdfX_mult( o3, r3, c1 );
10001- stbir__simdfX_store( output1, o0 ); stbir__simdfX_store( output1+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output1+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output1+(3*stbir__simdfX_float_count), o3 ); )
10002- stbIF2( stbir__simdfX_mult( o0, r0, c2 ); stbir__simdfX_mult( o1, r1, c2 ); stbir__simdfX_mult( o2, r2, c2 ); stbir__simdfX_mult( o3, r3, c2 );
10003- stbir__simdfX_store( output2, o0 ); stbir__simdfX_store( output2+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output2+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output2+(3*stbir__simdfX_float_count), o3 ); )
10004- stbIF3( stbir__simdfX_mult( o0, r0, c3 ); stbir__simdfX_mult( o1, r1, c3 ); stbir__simdfX_mult( o2, r2, c3 ); stbir__simdfX_mult( o3, r3, c3 );
10005- stbir__simdfX_store( output3, o0 ); stbir__simdfX_store( output3+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output3+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output3+(3*stbir__simdfX_float_count), o3 ); )
10006- stbIF4( stbir__simdfX_mult( o0, r0, c4 ); stbir__simdfX_mult( o1, r1, c4 ); stbir__simdfX_mult( o2, r2, c4 ); stbir__simdfX_mult( o3, r3, c4 );
10007- stbir__simdfX_store( output4, o0 ); stbir__simdfX_store( output4+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output4+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output4+(3*stbir__simdfX_float_count), o3 ); )
10008- stbIF5( stbir__simdfX_mult( o0, r0, c5 ); stbir__simdfX_mult( o1, r1, c5 ); stbir__simdfX_mult( o2, r2, c5 ); stbir__simdfX_mult( o3, r3, c5 );
10009- stbir__simdfX_store( output5, o0 ); stbir__simdfX_store( output5+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output5+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output5+(3*stbir__simdfX_float_count), o3 ); )
10010- stbIF6( stbir__simdfX_mult( o0, r0, c6 ); stbir__simdfX_mult( o1, r1, c6 ); stbir__simdfX_mult( o2, r2, c6 ); stbir__simdfX_mult( o3, r3, c6 );
10011- stbir__simdfX_store( output6, o0 ); stbir__simdfX_store( output6+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output6+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output6+(3*stbir__simdfX_float_count), o3 ); )
10012- stbIF7( stbir__simdfX_mult( o0, r0, c7 ); stbir__simdfX_mult( o1, r1, c7 ); stbir__simdfX_mult( o2, r2, c7 ); stbir__simdfX_mult( o3, r3, c7 );
10013- stbir__simdfX_store( output7, o0 ); stbir__simdfX_store( output7+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output7+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output7+(3*stbir__simdfX_float_count), o3 ); )
10014- #endif
10015-
10016- input += (4*stbir__simdfX_float_count);
10017- stbIF0( output0 += (4*stbir__simdfX_float_count); ) stbIF1( output1 += (4*stbir__simdfX_float_count); ) stbIF2( output2 += (4*stbir__simdfX_float_count); ) stbIF3( output3 += (4*stbir__simdfX_float_count); ) stbIF4( output4 += (4*stbir__simdfX_float_count); ) stbIF5( output5 += (4*stbir__simdfX_float_count); ) stbIF6( output6 += (4*stbir__simdfX_float_count); ) stbIF7( output7 += (4*stbir__simdfX_float_count); )
10018- }
10019- STBIR_SIMD_NO_UNROLL_LOOP_START
10020- while ( ( (char*)input_end - (char*) input ) >= 16 )
10021- {
10022- stbir__simdf o0, r0;
10023- STBIR_SIMD_NO_UNROLL(output0);
10024-
10025- stbir__simdf_load( r0, input );
10026-
10027- #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10028- stbIF0( stbir__simdf_load( o0, output0 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c0 ) ); stbir__simdf_store( output0, o0 ); )
10029- stbIF1( stbir__simdf_load( o0, output1 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c1 ) ); stbir__simdf_store( output1, o0 ); )
10030- stbIF2( stbir__simdf_load( o0, output2 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c2 ) ); stbir__simdf_store( output2, o0 ); )
10031- stbIF3( stbir__simdf_load( o0, output3 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c3 ) ); stbir__simdf_store( output3, o0 ); )
10032- stbIF4( stbir__simdf_load( o0, output4 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c4 ) ); stbir__simdf_store( output4, o0 ); )
10033- stbIF5( stbir__simdf_load( o0, output5 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c5 ) ); stbir__simdf_store( output5, o0 ); )
10034- stbIF6( stbir__simdf_load( o0, output6 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c6 ) ); stbir__simdf_store( output6, o0 ); )
10035- stbIF7( stbir__simdf_load( o0, output7 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c7 ) ); stbir__simdf_store( output7, o0 ); )
10036- #else
10037- stbIF0( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c0 ) ); stbir__simdf_store( output0, o0 ); )
10038- stbIF1( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c1 ) ); stbir__simdf_store( output1, o0 ); )
10039- stbIF2( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c2 ) ); stbir__simdf_store( output2, o0 ); )
10040- stbIF3( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c3 ) ); stbir__simdf_store( output3, o0 ); )
10041- stbIF4( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c4 ) ); stbir__simdf_store( output4, o0 ); )
10042- stbIF5( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c5 ) ); stbir__simdf_store( output5, o0 ); )
10043- stbIF6( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c6 ) ); stbir__simdf_store( output6, o0 ); )
10044- stbIF7( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c7 ) ); stbir__simdf_store( output7, o0 ); )
10045- #endif
10046-
10047- input += 4;
10048- stbIF0( output0 += 4; ) stbIF1( output1 += 4; ) stbIF2( output2 += 4; ) stbIF3( output3 += 4; ) stbIF4( output4 += 4; ) stbIF5( output5 += 4; ) stbIF6( output6 += 4; ) stbIF7( output7 += 4; )
10049- }
10050- }
10051- #else
10052- STBIR_NO_UNROLL_LOOP_START
10053- while ( ( (char*)input_end - (char*) input ) >= 16 )
10054- {
10055- float r0, r1, r2, r3;
10056- STBIR_NO_UNROLL(input);
10057-
10058- r0 = input[0], r1 = input[1], r2 = input[2], r3 = input[3];
10059-
10060- #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10061- stbIF0( output0[0] += ( r0 * c0s ); output0[1] += ( r1 * c0s ); output0[2] += ( r2 * c0s ); output0[3] += ( r3 * c0s ); )
10062- stbIF1( output1[0] += ( r0 * c1s ); output1[1] += ( r1 * c1s ); output1[2] += ( r2 * c1s ); output1[3] += ( r3 * c1s ); )
10063- stbIF2( output2[0] += ( r0 * c2s ); output2[1] += ( r1 * c2s ); output2[2] += ( r2 * c2s ); output2[3] += ( r3 * c2s ); )
10064- stbIF3( output3[0] += ( r0 * c3s ); output3[1] += ( r1 * c3s ); output3[2] += ( r2 * c3s ); output3[3] += ( r3 * c3s ); )
10065- stbIF4( output4[0] += ( r0 * c4s ); output4[1] += ( r1 * c4s ); output4[2] += ( r2 * c4s ); output4[3] += ( r3 * c4s ); )
10066- stbIF5( output5[0] += ( r0 * c5s ); output5[1] += ( r1 * c5s ); output5[2] += ( r2 * c5s ); output5[3] += ( r3 * c5s ); )
10067- stbIF6( output6[0] += ( r0 * c6s ); output6[1] += ( r1 * c6s ); output6[2] += ( r2 * c6s ); output6[3] += ( r3 * c6s ); )
10068- stbIF7( output7[0] += ( r0 * c7s ); output7[1] += ( r1 * c7s ); output7[2] += ( r2 * c7s ); output7[3] += ( r3 * c7s ); )
10069- #else
10070- stbIF0( output0[0] = ( r0 * c0s ); output0[1] = ( r1 * c0s ); output0[2] = ( r2 * c0s ); output0[3] = ( r3 * c0s ); )
10071- stbIF1( output1[0] = ( r0 * c1s ); output1[1] = ( r1 * c1s ); output1[2] = ( r2 * c1s ); output1[3] = ( r3 * c1s ); )
10072- stbIF2( output2[0] = ( r0 * c2s ); output2[1] = ( r1 * c2s ); output2[2] = ( r2 * c2s ); output2[3] = ( r3 * c2s ); )
10073- stbIF3( output3[0] = ( r0 * c3s ); output3[1] = ( r1 * c3s ); output3[2] = ( r2 * c3s ); output3[3] = ( r3 * c3s ); )
10074- stbIF4( output4[0] = ( r0 * c4s ); output4[1] = ( r1 * c4s ); output4[2] = ( r2 * c4s ); output4[3] = ( r3 * c4s ); )
10075- stbIF5( output5[0] = ( r0 * c5s ); output5[1] = ( r1 * c5s ); output5[2] = ( r2 * c5s ); output5[3] = ( r3 * c5s ); )
10076- stbIF6( output6[0] = ( r0 * c6s ); output6[1] = ( r1 * c6s ); output6[2] = ( r2 * c6s ); output6[3] = ( r3 * c6s ); )
10077- stbIF7( output7[0] = ( r0 * c7s ); output7[1] = ( r1 * c7s ); output7[2] = ( r2 * c7s ); output7[3] = ( r3 * c7s ); )
10078- #endif
10079-
10080- input += 4;
10081- stbIF0( output0 += 4; ) stbIF1( output1 += 4; ) stbIF2( output2 += 4; ) stbIF3( output3 += 4; ) stbIF4( output4 += 4; ) stbIF5( output5 += 4; ) stbIF6( output6 += 4; ) stbIF7( output7 += 4; )
10082- }
10083- #endif
10084- STBIR_NO_UNROLL_LOOP_START
10085- while ( input < input_end )
10086- {
10087- float r = input[0];
10088- STBIR_NO_UNROLL(output0);
10089-
10090- #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10091- stbIF0( output0[0] += ( r * c0s ); )
10092- stbIF1( output1[0] += ( r * c1s ); )
10093- stbIF2( output2[0] += ( r * c2s ); )
10094- stbIF3( output3[0] += ( r * c3s ); )
10095- stbIF4( output4[0] += ( r * c4s ); )
10096- stbIF5( output5[0] += ( r * c5s ); )
10097- stbIF6( output6[0] += ( r * c6s ); )
10098- stbIF7( output7[0] += ( r * c7s ); )
10099- #else
10100- stbIF0( output0[0] = ( r * c0s ); )
10101- stbIF1( output1[0] = ( r * c1s ); )
10102- stbIF2( output2[0] = ( r * c2s ); )
10103- stbIF3( output3[0] = ( r * c3s ); )
10104- stbIF4( output4[0] = ( r * c4s ); )
10105- stbIF5( output5[0] = ( r * c5s ); )
10106- stbIF6( output6[0] = ( r * c6s ); )
10107- stbIF7( output7[0] = ( r * c7s ); )
10108- #endif
10109-
10110- ++input;
10111- stbIF0( ++output0; ) stbIF1( ++output1; ) stbIF2( ++output2; ) stbIF3( ++output3; ) stbIF4( ++output4; ) stbIF5( ++output5; ) stbIF6( ++output6; ) stbIF7( ++output7; )
10112- }
10113-}
10114-
10115-static void STBIR_chans( stbir__vertical_gather_with_,_coeffs)( float * outputp, float const * vertical_coefficients, float const ** inputs, float const * input0_end )
10116-{
10117- float STBIR_SIMD_STREAMOUT_PTR( * ) output = outputp;
10118-
10119- stbIF0( float const * input0 = inputs[0]; float c0s = vertical_coefficients[0]; )
10120- stbIF1( float const * input1 = inputs[1]; float c1s = vertical_coefficients[1]; )
10121- stbIF2( float const * input2 = inputs[2]; float c2s = vertical_coefficients[2]; )
10122- stbIF3( float const * input3 = inputs[3]; float c3s = vertical_coefficients[3]; )
10123- stbIF4( float const * input4 = inputs[4]; float c4s = vertical_coefficients[4]; )
10124- stbIF5( float const * input5 = inputs[5]; float c5s = vertical_coefficients[5]; )
10125- stbIF6( float const * input6 = inputs[6]; float c6s = vertical_coefficients[6]; )
10126- stbIF7( float const * input7 = inputs[7]; float c7s = vertical_coefficients[7]; )
10127-
10128-#if ( STBIR__vertical_channels == 1 ) && !defined(STB_IMAGE_RESIZE_VERTICAL_CONTINUE)
10129- // check single channel one weight
10130- if ( ( c0s >= (1.0f-0.000001f) ) && ( c0s <= (1.0f+0.000001f) ) )
10131- {
10132- STBIR_MEMCPY( output, input0, (char*)input0_end - (char*)input0 );
10133- return;
10134- }
10135-#endif
10136-
10137- #ifdef STBIR_SIMD
10138- {
10139- stbIF0(stbir__simdfX c0 = stbir__simdf_frepX( c0s ); )
10140- stbIF1(stbir__simdfX c1 = stbir__simdf_frepX( c1s ); )
10141- stbIF2(stbir__simdfX c2 = stbir__simdf_frepX( c2s ); )
10142- stbIF3(stbir__simdfX c3 = stbir__simdf_frepX( c3s ); )
10143- stbIF4(stbir__simdfX c4 = stbir__simdf_frepX( c4s ); )
10144- stbIF5(stbir__simdfX c5 = stbir__simdf_frepX( c5s ); )
10145- stbIF6(stbir__simdfX c6 = stbir__simdf_frepX( c6s ); )
10146- stbIF7(stbir__simdfX c7 = stbir__simdf_frepX( c7s ); )
10147-
10148- STBIR_SIMD_NO_UNROLL_LOOP_START
10149- while ( ( (char*)input0_end - (char*) input0 ) >= (16*stbir__simdfX_float_count) )
10150- {
10151- stbir__simdfX o0, o1, o2, o3, r0, r1, r2, r3;
10152- STBIR_SIMD_NO_UNROLL(output);
10153-
10154- // prefetch four loop iterations ahead (doesn't affect much for small resizes, but helps with big ones)
10155- stbIF0( stbir__prefetch( input0 + (16*stbir__simdfX_float_count) ); )
10156- stbIF1( stbir__prefetch( input1 + (16*stbir__simdfX_float_count) ); )
10157- stbIF2( stbir__prefetch( input2 + (16*stbir__simdfX_float_count) ); )
10158- stbIF3( stbir__prefetch( input3 + (16*stbir__simdfX_float_count) ); )
10159- stbIF4( stbir__prefetch( input4 + (16*stbir__simdfX_float_count) ); )
10160- stbIF5( stbir__prefetch( input5 + (16*stbir__simdfX_float_count) ); )
10161- stbIF6( stbir__prefetch( input6 + (16*stbir__simdfX_float_count) ); )
10162- stbIF7( stbir__prefetch( input7 + (16*stbir__simdfX_float_count) ); )
10163-
10164- #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10165- stbIF0( stbir__simdfX_load( o0, output ); stbir__simdfX_load( o1, output+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output+(3*stbir__simdfX_float_count) );
10166- stbir__simdfX_load( r0, input0 ); stbir__simdfX_load( r1, input0+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input0+(3*stbir__simdfX_float_count) );
10167- stbir__simdfX_madd( o0, o0, r0, c0 ); stbir__simdfX_madd( o1, o1, r1, c0 ); stbir__simdfX_madd( o2, o2, r2, c0 ); stbir__simdfX_madd( o3, o3, r3, c0 ); )
10168- #else
10169- stbIF0( stbir__simdfX_load( r0, input0 ); stbir__simdfX_load( r1, input0+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input0+(3*stbir__simdfX_float_count) );
10170- stbir__simdfX_mult( o0, r0, c0 ); stbir__simdfX_mult( o1, r1, c0 ); stbir__simdfX_mult( o2, r2, c0 ); stbir__simdfX_mult( o3, r3, c0 ); )
10171- #endif
10172-
10173- stbIF1( stbir__simdfX_load( r0, input1 ); stbir__simdfX_load( r1, input1+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input1+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input1+(3*stbir__simdfX_float_count) );
10174- stbir__simdfX_madd( o0, o0, r0, c1 ); stbir__simdfX_madd( o1, o1, r1, c1 ); stbir__simdfX_madd( o2, o2, r2, c1 ); stbir__simdfX_madd( o3, o3, r3, c1 ); )
10175- stbIF2( stbir__simdfX_load( r0, input2 ); stbir__simdfX_load( r1, input2+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input2+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input2+(3*stbir__simdfX_float_count) );
10176- stbir__simdfX_madd( o0, o0, r0, c2 ); stbir__simdfX_madd( o1, o1, r1, c2 ); stbir__simdfX_madd( o2, o2, r2, c2 ); stbir__simdfX_madd( o3, o3, r3, c2 ); )
10177- stbIF3( stbir__simdfX_load( r0, input3 ); stbir__simdfX_load( r1, input3+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input3+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input3+(3*stbir__simdfX_float_count) );
10178- stbir__simdfX_madd( o0, o0, r0, c3 ); stbir__simdfX_madd( o1, o1, r1, c3 ); stbir__simdfX_madd( o2, o2, r2, c3 ); stbir__simdfX_madd( o3, o3, r3, c3 ); )
10179- stbIF4( stbir__simdfX_load( r0, input4 ); stbir__simdfX_load( r1, input4+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input4+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input4+(3*stbir__simdfX_float_count) );
10180- stbir__simdfX_madd( o0, o0, r0, c4 ); stbir__simdfX_madd( o1, o1, r1, c4 ); stbir__simdfX_madd( o2, o2, r2, c4 ); stbir__simdfX_madd( o3, o3, r3, c4 ); )
10181- stbIF5( stbir__simdfX_load( r0, input5 ); stbir__simdfX_load( r1, input5+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input5+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input5+(3*stbir__simdfX_float_count) );
10182- stbir__simdfX_madd( o0, o0, r0, c5 ); stbir__simdfX_madd( o1, o1, r1, c5 ); stbir__simdfX_madd( o2, o2, r2, c5 ); stbir__simdfX_madd( o3, o3, r3, c5 ); )
10183- stbIF6( stbir__simdfX_load( r0, input6 ); stbir__simdfX_load( r1, input6+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input6+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input6+(3*stbir__simdfX_float_count) );
10184- stbir__simdfX_madd( o0, o0, r0, c6 ); stbir__simdfX_madd( o1, o1, r1, c6 ); stbir__simdfX_madd( o2, o2, r2, c6 ); stbir__simdfX_madd( o3, o3, r3, c6 ); )
10185- stbIF7( stbir__simdfX_load( r0, input7 ); stbir__simdfX_load( r1, input7+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input7+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input7+(3*stbir__simdfX_float_count) );
10186- stbir__simdfX_madd( o0, o0, r0, c7 ); stbir__simdfX_madd( o1, o1, r1, c7 ); stbir__simdfX_madd( o2, o2, r2, c7 ); stbir__simdfX_madd( o3, o3, r3, c7 ); )
10187-
10188- stbir__simdfX_store( output, o0 ); stbir__simdfX_store( output+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output+(3*stbir__simdfX_float_count), o3 );
10189- output += (4*stbir__simdfX_float_count);
10190- stbIF0( input0 += (4*stbir__simdfX_float_count); ) stbIF1( input1 += (4*stbir__simdfX_float_count); ) stbIF2( input2 += (4*stbir__simdfX_float_count); ) stbIF3( input3 += (4*stbir__simdfX_float_count); ) stbIF4( input4 += (4*stbir__simdfX_float_count); ) stbIF5( input5 += (4*stbir__simdfX_float_count); ) stbIF6( input6 += (4*stbir__simdfX_float_count); ) stbIF7( input7 += (4*stbir__simdfX_float_count); )
10191- }
10192-
10193- STBIR_SIMD_NO_UNROLL_LOOP_START
10194- while ( ( (char*)input0_end - (char*) input0 ) >= 16 )
10195- {
10196- stbir__simdf o0, r0;
10197- STBIR_SIMD_NO_UNROLL(output);
10198-
10199- #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10200- stbIF0( stbir__simdf_load( o0, output ); stbir__simdf_load( r0, input0 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c0 ) ); )
10201- #else
10202- stbIF0( stbir__simdf_load( r0, input0 ); stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c0 ) ); )
10203- #endif
10204- stbIF1( stbir__simdf_load( r0, input1 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c1 ) ); )
10205- stbIF2( stbir__simdf_load( r0, input2 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c2 ) ); )
10206- stbIF3( stbir__simdf_load( r0, input3 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c3 ) ); )
10207- stbIF4( stbir__simdf_load( r0, input4 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c4 ) ); )
10208- stbIF5( stbir__simdf_load( r0, input5 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c5 ) ); )
10209- stbIF6( stbir__simdf_load( r0, input6 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c6 ) ); )
10210- stbIF7( stbir__simdf_load( r0, input7 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c7 ) ); )
10211-
10212- stbir__simdf_store( output, o0 );
10213- output += 4;
10214- stbIF0( input0 += 4; ) stbIF1( input1 += 4; ) stbIF2( input2 += 4; ) stbIF3( input3 += 4; ) stbIF4( input4 += 4; ) stbIF5( input5 += 4; ) stbIF6( input6 += 4; ) stbIF7( input7 += 4; )
10215- }
10216- }
10217- #else
10218- STBIR_NO_UNROLL_LOOP_START
10219- while ( ( (char*)input0_end - (char*) input0 ) >= 16 )
10220- {
10221- float o0, o1, o2, o3;
10222- STBIR_NO_UNROLL(output);
10223- #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10224- stbIF0( o0 = output[0] + input0[0] * c0s; o1 = output[1] + input0[1] * c0s; o2 = output[2] + input0[2] * c0s; o3 = output[3] + input0[3] * c0s; )
10225- #else
10226- stbIF0( o0 = input0[0] * c0s; o1 = input0[1] * c0s; o2 = input0[2] * c0s; o3 = input0[3] * c0s; )
10227- #endif
10228- stbIF1( o0 += input1[0] * c1s; o1 += input1[1] * c1s; o2 += input1[2] * c1s; o3 += input1[3] * c1s; )
10229- stbIF2( o0 += input2[0] * c2s; o1 += input2[1] * c2s; o2 += input2[2] * c2s; o3 += input2[3] * c2s; )
10230- stbIF3( o0 += input3[0] * c3s; o1 += input3[1] * c3s; o2 += input3[2] * c3s; o3 += input3[3] * c3s; )
10231- stbIF4( o0 += input4[0] * c4s; o1 += input4[1] * c4s; o2 += input4[2] * c4s; o3 += input4[3] * c4s; )
10232- stbIF5( o0 += input5[0] * c5s; o1 += input5[1] * c5s; o2 += input5[2] * c5s; o3 += input5[3] * c5s; )
10233- stbIF6( o0 += input6[0] * c6s; o1 += input6[1] * c6s; o2 += input6[2] * c6s; o3 += input6[3] * c6s; )
10234- stbIF7( o0 += input7[0] * c7s; o1 += input7[1] * c7s; o2 += input7[2] * c7s; o3 += input7[3] * c7s; )
10235- output[0] = o0; output[1] = o1; output[2] = o2; output[3] = o3;
10236- output += 4;
10237- stbIF0( input0 += 4; ) stbIF1( input1 += 4; ) stbIF2( input2 += 4; ) stbIF3( input3 += 4; ) stbIF4( input4 += 4; ) stbIF5( input5 += 4; ) stbIF6( input6 += 4; ) stbIF7( input7 += 4; )
10238- }
10239- #endif
10240- STBIR_NO_UNROLL_LOOP_START
10241- while ( input0 < input0_end )
10242- {
10243- float o0;
10244- STBIR_NO_UNROLL(output);
10245- #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10246- stbIF0( o0 = output[0] + input0[0] * c0s; )
10247- #else
10248- stbIF0( o0 = input0[0] * c0s; )
10249- #endif
10250- stbIF1( o0 += input1[0] * c1s; )
10251- stbIF2( o0 += input2[0] * c2s; )
10252- stbIF3( o0 += input3[0] * c3s; )
10253- stbIF4( o0 += input4[0] * c4s; )
10254- stbIF5( o0 += input5[0] * c5s; )
10255- stbIF6( o0 += input6[0] * c6s; )
10256- stbIF7( o0 += input7[0] * c7s; )
10257- output[0] = o0;
10258- ++output;
10259- stbIF0( ++input0; ) stbIF1( ++input1; ) stbIF2( ++input2; ) stbIF3( ++input3; ) stbIF4( ++input4; ) stbIF5( ++input5; ) stbIF6( ++input6; ) stbIF7( ++input7; )
10260- }
10261-}
10262-
10263-#undef stbIF0
10264-#undef stbIF1
10265-#undef stbIF2
10266-#undef stbIF3
10267-#undef stbIF4
10268-#undef stbIF5
10269-#undef stbIF6
10270-#undef stbIF7
10271-#undef STB_IMAGE_RESIZE_DO_VERTICALS
10272-#undef STBIR__vertical_channels
10273-#undef STB_IMAGE_RESIZE_DO_HORIZONTALS
10274-#undef STBIR_strs_join24
10275-#undef STBIR_strs_join14
10276-#undef STBIR_chans
10277-#ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10278-#undef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10279-#endif
10280-
10281-#else // !STB_IMAGE_RESIZE_DO_VERTICALS
10282-
10283-#define STBIR_chans( start, end ) STBIR_strs_join1(start,STBIR__horizontal_channels,end)
10284-
10285-#ifndef stbir__2_coeff_only
10286-#define stbir__2_coeff_only() \
10287- stbir__1_coeff_only(); \
10288- stbir__1_coeff_remnant(1);
10289-#endif
10290-
10291-#ifndef stbir__2_coeff_remnant
10292-#define stbir__2_coeff_remnant( ofs ) \
10293- stbir__1_coeff_remnant(ofs); \
10294- stbir__1_coeff_remnant((ofs)+1);
10295-#endif
10296-
10297-#ifndef stbir__3_coeff_only
10298-#define stbir__3_coeff_only() \
10299- stbir__2_coeff_only(); \
10300- stbir__1_coeff_remnant(2);
10301-#endif
10302-
10303-#ifndef stbir__3_coeff_remnant
10304-#define stbir__3_coeff_remnant( ofs ) \
10305- stbir__2_coeff_remnant(ofs); \
10306- stbir__1_coeff_remnant((ofs)+2);
10307-#endif
10308-
10309-#ifndef stbir__3_coeff_setup
10310-#define stbir__3_coeff_setup()
10311-#endif
10312-
10313-#ifndef stbir__4_coeff_start
10314-#define stbir__4_coeff_start() \
10315- stbir__2_coeff_only(); \
10316- stbir__2_coeff_remnant(2);
10317-#endif
10318-
10319-#ifndef stbir__4_coeff_continue_from_4
10320-#define stbir__4_coeff_continue_from_4( ofs ) \
10321- stbir__2_coeff_remnant(ofs); \
10322- stbir__2_coeff_remnant((ofs)+2);
10323-#endif
10324-
10325-#ifndef stbir__store_output_tiny
10326-#define stbir__store_output_tiny stbir__store_output
10327-#endif
10328-
10329-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_1_coeff)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10330-{
10331- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10332- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10333- STBIR_SIMD_NO_UNROLL_LOOP_START
10334- do {
10335- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10336- float const * hc = horizontal_coefficients;
10337- stbir__1_coeff_only();
10338- stbir__store_output_tiny();
10339- } while ( output < output_end );
10340-}
10341-
10342-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_2_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10343-{
10344- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10345- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10346- STBIR_SIMD_NO_UNROLL_LOOP_START
10347- do {
10348- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10349- float const * hc = horizontal_coefficients;
10350- stbir__2_coeff_only();
10351- stbir__store_output_tiny();
10352- } while ( output < output_end );
10353-}
10354-
10355-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_3_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10356-{
10357- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10358- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10359- STBIR_SIMD_NO_UNROLL_LOOP_START
10360- do {
10361- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10362- float const * hc = horizontal_coefficients;
10363- stbir__3_coeff_only();
10364- stbir__store_output_tiny();
10365- } while ( output < output_end );
10366-}
10367-
10368-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_4_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10369-{
10370- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10371- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10372- STBIR_SIMD_NO_UNROLL_LOOP_START
10373- do {
10374- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10375- float const * hc = horizontal_coefficients;
10376- stbir__4_coeff_start();
10377- stbir__store_output();
10378- } while ( output < output_end );
10379-}
10380-
10381-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_5_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10382-{
10383- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10384- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10385- STBIR_SIMD_NO_UNROLL_LOOP_START
10386- do {
10387- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10388- float const * hc = horizontal_coefficients;
10389- stbir__4_coeff_start();
10390- stbir__1_coeff_remnant(4);
10391- stbir__store_output();
10392- } while ( output < output_end );
10393-}
10394-
10395-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_6_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10396-{
10397- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10398- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10399- STBIR_SIMD_NO_UNROLL_LOOP_START
10400- do {
10401- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10402- float const * hc = horizontal_coefficients;
10403- stbir__4_coeff_start();
10404- stbir__2_coeff_remnant(4);
10405- stbir__store_output();
10406- } while ( output < output_end );
10407-}
10408-
10409-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_7_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10410-{
10411- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10412- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10413- stbir__3_coeff_setup();
10414- STBIR_SIMD_NO_UNROLL_LOOP_START
10415- do {
10416- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10417- float const * hc = horizontal_coefficients;
10418-
10419- stbir__4_coeff_start();
10420- stbir__3_coeff_remnant(4);
10421- stbir__store_output();
10422- } while ( output < output_end );
10423-}
10424-
10425-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_8_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10426-{
10427- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10428- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10429- STBIR_SIMD_NO_UNROLL_LOOP_START
10430- do {
10431- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10432- float const * hc = horizontal_coefficients;
10433- stbir__4_coeff_start();
10434- stbir__4_coeff_continue_from_4(4);
10435- stbir__store_output();
10436- } while ( output < output_end );
10437-}
10438-
10439-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_9_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10440-{
10441- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10442- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10443- STBIR_SIMD_NO_UNROLL_LOOP_START
10444- do {
10445- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10446- float const * hc = horizontal_coefficients;
10447- stbir__4_coeff_start();
10448- stbir__4_coeff_continue_from_4(4);
10449- stbir__1_coeff_remnant(8);
10450- stbir__store_output();
10451- } while ( output < output_end );
10452-}
10453-
10454-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_10_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10455-{
10456- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10457- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10458- STBIR_SIMD_NO_UNROLL_LOOP_START
10459- do {
10460- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10461- float const * hc = horizontal_coefficients;
10462- stbir__4_coeff_start();
10463- stbir__4_coeff_continue_from_4(4);
10464- stbir__2_coeff_remnant(8);
10465- stbir__store_output();
10466- } while ( output < output_end );
10467-}
10468-
10469-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_11_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10470-{
10471- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10472- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10473- stbir__3_coeff_setup();
10474- STBIR_SIMD_NO_UNROLL_LOOP_START
10475- do {
10476- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10477- float const * hc = horizontal_coefficients;
10478- stbir__4_coeff_start();
10479- stbir__4_coeff_continue_from_4(4);
10480- stbir__3_coeff_remnant(8);
10481- stbir__store_output();
10482- } while ( output < output_end );
10483-}
10484-
10485-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_12_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10486-{
10487- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10488- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10489- STBIR_SIMD_NO_UNROLL_LOOP_START
10490- do {
10491- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10492- float const * hc = horizontal_coefficients;
10493- stbir__4_coeff_start();
10494- stbir__4_coeff_continue_from_4(4);
10495- stbir__4_coeff_continue_from_4(8);
10496- stbir__store_output();
10497- } while ( output < output_end );
10498-}
10499-
10500-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod0 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10501-{
10502- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10503- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10504- STBIR_SIMD_NO_UNROLL_LOOP_START
10505- do {
10506- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10507- int n = ( ( horizontal_contributors->n1 - horizontal_contributors->n0 + 1 ) - 4 + 3 ) >> 2;
10508- float const * hc = horizontal_coefficients;
10509-
10510- stbir__4_coeff_start();
10511- STBIR_SIMD_NO_UNROLL_LOOP_START
10512- do {
10513- hc += 4;
10514- decode += STBIR__horizontal_channels * 4;
10515- stbir__4_coeff_continue_from_4( 0 );
10516- --n;
10517- } while ( n > 0 );
10518- stbir__store_output();
10519- } while ( output < output_end );
10520-}
10521-
10522-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod1 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10523-{
10524- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10525- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10526- STBIR_SIMD_NO_UNROLL_LOOP_START
10527- do {
10528- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10529- int n = ( ( horizontal_contributors->n1 - horizontal_contributors->n0 + 1 ) - 5 + 3 ) >> 2;
10530- float const * hc = horizontal_coefficients;
10531-
10532- stbir__4_coeff_start();
10533- STBIR_SIMD_NO_UNROLL_LOOP_START
10534- do {
10535- hc += 4;
10536- decode += STBIR__horizontal_channels * 4;
10537- stbir__4_coeff_continue_from_4( 0 );
10538- --n;
10539- } while ( n > 0 );
10540- stbir__1_coeff_remnant( 4 );
10541- stbir__store_output();
10542- } while ( output < output_end );
10543-}
10544-
10545-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod2 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10546-{
10547- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10548- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10549- STBIR_SIMD_NO_UNROLL_LOOP_START
10550- do {
10551- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10552- int n = ( ( horizontal_contributors->n1 - horizontal_contributors->n0 + 1 ) - 6 + 3 ) >> 2;
10553- float const * hc = horizontal_coefficients;
10554-
10555- stbir__4_coeff_start();
10556- STBIR_SIMD_NO_UNROLL_LOOP_START
10557- do {
10558- hc += 4;
10559- decode += STBIR__horizontal_channels * 4;
10560- stbir__4_coeff_continue_from_4( 0 );
10561- --n;
10562- } while ( n > 0 );
10563- stbir__2_coeff_remnant( 4 );
10564-
10565- stbir__store_output();
10566- } while ( output < output_end );
10567-}
10568-
10569-static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod3 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10570-{
10571- float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10572- float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10573- stbir__3_coeff_setup();
10574- STBIR_SIMD_NO_UNROLL_LOOP_START
10575- do {
10576- float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10577- int n = ( ( horizontal_contributors->n1 - horizontal_contributors->n0 + 1 ) - 7 + 3 ) >> 2;
10578- float const * hc = horizontal_coefficients;
10579-
10580- stbir__4_coeff_start();
10581- STBIR_SIMD_NO_UNROLL_LOOP_START
10582- do {
10583- hc += 4;
10584- decode += STBIR__horizontal_channels * 4;
10585- stbir__4_coeff_continue_from_4( 0 );
10586- --n;
10587- } while ( n > 0 );
10588- stbir__3_coeff_remnant( 4 );
10589-
10590- stbir__store_output();
10591- } while ( output < output_end );
10592-}
10593-
10594-static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_funcs)[4]=
10595-{
10596- STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod0),
10597- STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod1),
10598- STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod2),
10599- STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod3),
10600-};
10601-
10602-static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_funcs)[12]=
10603-{
10604- STBIR_chans(stbir__horizontal_gather_,_channels_with_1_coeff),
10605- STBIR_chans(stbir__horizontal_gather_,_channels_with_2_coeffs),
10606- STBIR_chans(stbir__horizontal_gather_,_channels_with_3_coeffs),
10607- STBIR_chans(stbir__horizontal_gather_,_channels_with_4_coeffs),
10608- STBIR_chans(stbir__horizontal_gather_,_channels_with_5_coeffs),
10609- STBIR_chans(stbir__horizontal_gather_,_channels_with_6_coeffs),
10610- STBIR_chans(stbir__horizontal_gather_,_channels_with_7_coeffs),
10611- STBIR_chans(stbir__horizontal_gather_,_channels_with_8_coeffs),
10612- STBIR_chans(stbir__horizontal_gather_,_channels_with_9_coeffs),
10613- STBIR_chans(stbir__horizontal_gather_,_channels_with_10_coeffs),
10614- STBIR_chans(stbir__horizontal_gather_,_channels_with_11_coeffs),
10615- STBIR_chans(stbir__horizontal_gather_,_channels_with_12_coeffs),
10616-};
10617-
10618-#undef STBIR__horizontal_channels
10619-#undef STB_IMAGE_RESIZE_DO_HORIZONTALS
10620-#undef stbir__1_coeff_only
10621-#undef stbir__1_coeff_remnant
10622-#undef stbir__2_coeff_only
10623-#undef stbir__2_coeff_remnant
10624-#undef stbir__3_coeff_only
10625-#undef stbir__3_coeff_remnant
10626-#undef stbir__3_coeff_setup
10627-#undef stbir__4_coeff_start
10628-#undef stbir__4_coeff_continue_from_4
10629-#undef stbir__store_output
10630-#undef stbir__store_output_tiny
10631-#undef STBIR_chans
10632-
10633-#endif // HORIZONALS
10634-
10635-#undef STBIR_strs_join2
10636-#undef STBIR_strs_join1
10637-
10638-#endif // STB_IMAGE_RESIZE_DO_HORIZONTALS/VERTICALS/CODERS
10639-
10640-/*
10641-------------------------------------------------------------------------------
10642-This software is available under 2 licenses -- choose whichever you prefer.
10643-------------------------------------------------------------------------------
10644-ALTERNATIVE A - MIT License
10645-Copyright (c) 2017 Sean Barrett
10646-Permission is hereby granted, free of charge, to any person obtaining a copy of
10647-this software and associated documentation files (the "Software"), to deal in
10648-the Software without restriction, including without limitation the rights to
10649-use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
10650-of the Software, and to permit persons to whom the Software is furnished to do
10651-so, subject to the following conditions:
10652-The above copyright notice and this permission notice shall be included in all
10653-copies or substantial portions of the Software.
10654-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
10655-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
10656-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
10657-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
10658-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
10659-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
10660-SOFTWARE.
10661-------------------------------------------------------------------------------
10662-ALTERNATIVE B - Public Domain (www.unlicense.org)
10663-This is free and unencumbered software released into the public domain.
10664-Anyone is free to copy, modify, publish, use, compile, sell, or distribute this
10665-software, either in source code form or as a compiled binary, for any purpose,
10666-commercial or non-commercial, and by any means.
10667-In jurisdictions that recognize copyright laws, the author or authors of this
10668-software dedicate any and all copyright interest in the software to the public
10669-domain. We make this dedication for the benefit of the public at large and to
10670-the detriment of our heirs and successors. We intend this dedication to be an
10671-overt act of relinquishment in perpetuity of all present and future rights to
10672-this software under copyright law.
10673-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
10674-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
10675-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
10676-AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
10677-ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
10678-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10679-------------------------------------------------------------------------------
10680-*/
10681+/* stb_image_resize2 - v2.50 - public domain image resizing
10682+
10683+ by Jeff Roberts (v2) and Jorge L Rodriguez
10684+ http://github.com/nothings/stb
10685+
10686+ Can be threaded with the extended API. SSE2, AVX, Neon and WASM SIMD support. Only
10687+ scaling and translation is supported, no rotations or shears.
10688+
10689+ COMPILING & LINKING
10690+ In one C/C++ file that #includes this file, do this:
10691+ #define STB_IMAGE_RESIZE_IMPLEMENTATION
10692+ before the #include. That will create the implementation in that file.
10693+
10694+ EASY API CALLS:
10695+ Easy API downsamples w/Mitchell filter, upsamples w/cubic interpolation, clamps to edge.
10696+
10697+ stbir_resize_uint8_srgb( input_pixels, input_w, input_h, input_stride_in_bytes,
10698+ output_pixels, output_w, output_h, output_stride_in_bytes,
10699+ pixel_layout_enum )
10700+
10701+ stbir_resize_uint8_linear( input_pixels, input_w, input_h, input_stride_in_bytes,
10702+ output_pixels, output_w, output_h, output_stride_in_bytes,
10703+ pixel_layout_enum )
10704+
10705+ stbir_resize_float_linear( input_pixels, input_w, input_h, input_stride_in_bytes,
10706+ output_pixels, output_w, output_h, output_stride_in_bytes,
10707+ pixel_layout_enum )
10708+
10709+ If you pass NULL or zero for the output_pixels, we will allocate the output buffer
10710+ for you and return it from the function (free with free() or STBIR_FREE).
10711+ As a special case, XX_stride_in_bytes of 0 means packed continuously in memory.
10712+
10713+ API LEVELS
10714+ There are three levels of API - easy-to-use, medium-complexity and extended-complexity.
10715+
10716+ See the "header file" section of the source for API documentation.
10717+
10718+ ADDITIONAL DOCUMENTATION
10719+
10720+ MEMORY ALLOCATION
10721+ By default, we use malloc and free for memory allocation. To override the
10722+ memory allocation, before the implementation #include, add a:
10723+
10724+ #define STBIR_MALLOC(size,user_data) ...
10725+ #define STBIR_FREE(ptr,user_data) ...
10726+
10727+ Each resize makes exactly one call to malloc/free (unless you use the
10728+ extended API where you can do one allocation for many resizes). Under
10729+ address sanitizer, we do separate allocations to find overread/writes.
10730+
10731+ PERFORMANCE
10732+ This library was written with an emphasis on performance. When testing
10733+ stb_image_resize with RGBA, the fastest mode is STBIR_4CHANNEL with
10734+ STBIR_TYPE_UINT8 pixels and CLAMPed edges (which is what many other resize
10735+ libs do by default). Also, make sure SIMD is turned on of course (default
10736+ for 64-bit targets). Avoid WRAP edge mode if you want the fastest speed.
10737+
10738+ This library also comes with profiling built-in. If you define STBIR_PROFILE,
10739+ you can use the advanced API and get low-level profiling information by
10740+ calling stbir_resize_extended_profile_info() or stbir_resize_split_profile_info()
10741+ after a resize.
10742+
10743+ SIMD
10744+ Most of the routines have optimized SSE2, AVX, AVX-512, NEON and WASM versions.
10745+
10746+ On Microsoft compilers, we automatically turn on SIMD for 64-bit x64 and
10747+ ARM; for 32-bit x86 and ARM, you select SIMD mode by defining STBIR_SSE2 or
10748+ STBIR_NEON.
10749+
10750+ For AVX and AVX2, we auto-select it by detecting the /arch:AVX
10751+ or /arch:AVX2 switches. You can also always manually turn SSE2, AVX or AVX2
10752+ support on by defining STBIR_SSE2, STBIR_AVX or STBIR_AVX2.
10753+
10754+ For AVX-512, we turn it on by /arch:AVX512 (or -march=x86-64-v4, or the
10755+ -mavx512f -mavx512bw -mavx512vl trio on clang and GCC), or by defining
10756+ STBIR_AVX512. It's mostly useful on Zen 5 or modern Intel cores that don't
10757+ downclock on use. Best on very large resizes.
10758+
10759+ On Linux, SSE2 and Neon is on by default for 64-bit x64 or ARM64. For 32-bit,
10760+ we select x86 SIMD mode by whether you have -msse2, -mavx or -mavx2 enabled
10761+ on the command line. For 32-bit ARM, you must pass -mfpu=neon-vfpv4 for both
10762+ clang and GCC, but GCC also requires an additional -mfp16-format=ieee to
10763+ automatically enable NEON.
10764+
10765+ On x86 platforms, you can also define STBIR_FP16C to turn on FP16C instructions
10766+ for converting back and forth to half-floats. This is autoselected when we
10767+ are using AVX2. Clang and GCC also require the -mf16c switch. ARM always uses
10768+ the built-in half float hardware NEON instructions.
10769+
10770+ You can also tell us to use multiply-add instructions with STBIR_USE_FMA.
10771+ Because x86 doesn't always have fma, we turn it off by default to maintain
10772+ determinism across all platforms. If you don't care about non-FMA determinism
10773+ and are willing to restrict yourself to more recent x86 CPUs (around the AVX
10774+ timeframe), then fma will give you around a 15% speedup.
10775+
10776+ You can force off SIMD in all cases by defining STBIR_NO_SIMD. You can turn
10777+ off AVX, AVX2 or AVX-512 specifically with STBIR_NO_AVX, STBIR_NO_AVX2 or
10778+ STBIR_NO_AVX512. AVX is 10% to 40% faster, and AVX2 is generally another 12%.
10779+
10780+ ALPHA CHANNEL
10781+ Most of the resizing functions provide the ability to control how the alpha
10782+ channel of an image is processed.
10783+
10784+ When alpha represents transparency, it is important that when combining
10785+ colors with filtering, the pixels should not be treated equally; they
10786+ should use a weighted average based on their alpha values. For example,
10787+ if a pixel is 1% opaque bright green and another pixel is 99% opaque
10788+ black and you average them, the average will be 50% opaque, but the
10789+ unweighted average and will be a middling green color, while the weighted
10790+ average will be nearly black. This means the unweighted version introduced
10791+ green energy that didn't exist in the source image.
10792+
10793+ (If you want to know why this makes sense, you can work out the math for
10794+ the following: consider what happens if you alpha composite a source image
10795+ over a fixed color and then average the output, vs. if you average the
10796+ source image pixels and then composite that over the same fixed color.
10797+ Only the weighted average produces the same result as the ground truth
10798+ composite-then-average result.)
10799+
10800+ Therefore, it is in general best to "alpha weight" the pixels when applying
10801+ filters to them. This essentially means multiplying the colors by the alpha
10802+ values before combining them, and then dividing by the alpha value at the
10803+ end.
10804+
10805+ The computer graphics industry introduced a technique called "premultiplied
10806+ alpha" or "associated alpha" in which image colors are stored in image files
10807+ already multiplied by their alpha. This saves some math when compositing,
10808+ and also avoids the need to divide by the alpha at the end (which is quite
10809+ inefficient). However, while premultiplied alpha is common in the movie CGI
10810+ industry, it is not commonplace in other industries like videogames, and most
10811+ consumer file formats are generally expected to contain not-premultiplied
10812+ colors. For example, Photoshop saves PNG files "unpremultiplied", and web
10813+ browsers like Chrome and Firefox expect PNG images to be unpremultiplied.
10814+
10815+ Note that there are three possibilities that might describe your image
10816+ and resize expectation:
10817+
10818+ 1. images are not premultiplied, alpha weighting is desired
10819+ 2. images are not premultiplied, alpha weighting is not desired
10820+ 3. images are premultiplied
10821+
10822+ Both case #2 and case #3 require the exact same math: no alpha weighting
10823+ should be applied or removed. Only case 1 requires extra math operations;
10824+ the other two cases can be handled identically.
10825+
10826+ stb_image_resize expects case #1 by default, applying alpha weighting to
10827+ images, expecting the input images to be unpremultiplied. This is what the
10828+ COLOR+ALPHA buffer types tell the resizer to do.
10829+
10830+ When you use the pixel layouts STBIR_RGBA, STBIR_BGRA, STBIR_ARGB,
10831+ STBIR_ABGR, STBIR_RA, or STBIR_AR you are telling us that the pixels are
10832+ non-premultiplied. In these cases, the resizer will alpha weight the colors
10833+ (effectively creating the premultiplied image), do the filtering, and then
10834+ convert back to non-premult on exit.
10835+
10836+ When you use the pixel layouts STBIR_RGBA_PM, STBIR_BGRA_PM, STBIR_ARGB_PM,
10837+ STBIR_ABGR_PM, STBIR_RA_PM or STBIR_AR_PM, you are telling that the pixels
10838+ ARE premultiplied. In this case, the resizer doesn't have to do the
10839+ premultipling - it can filter directly on the input. This about twice as
10840+ fast as the non-premultiplied case, so it's the right option if your data is
10841+ already setup correctly.
10842+
10843+ When you use the pixel layout STBIR_4CHANNEL or STBIR_2CHANNEL, you are
10844+ telling us that there is no channel that represents transparency; it may be
10845+ RGB and some unrelated fourth channel that has been stored in the alpha
10846+ channel, but it is actually not alpha. No special processing will be
10847+ performed.
10848+
10849+ The difference between the generic 4 or 2 channel layouts, and the
10850+ specialized _PM versions is with the _PM versions you are telling us that
10851+ the data *is* alpha, just don't premultiply it. That's important when
10852+ using SRGB pixel formats, we need to know where the alpha is, because
10853+ it is converted linearly (rather than with the SRGB converters).
10854+
10855+ Because alpha weighting produces the same effect as premultiplying, you
10856+ even have the option with non-premultiplied inputs to let the resizer
10857+ produce a premultiplied output. Because the intially computed alpha-weighted
10858+ output image is effectively premultiplied, this is actually more performant
10859+ than the normal path which un-premultiplies the output image as a final step.
10860+
10861+ Finally, when converting both in and out of non-premulitplied space (for
10862+ example, when using STBIR_RGBA), we go to somewhat heroic measures to
10863+ ensure that areas with zero alpha value pixels get something reasonable
10864+ in the RGB values. If you don't care about the RGB values of zero alpha
10865+ pixels, you can call the stbir_set_non_pm_alpha_speed_over_quality()
10866+ function - this runs a premultiplied resize about 25% faster. That said,
10867+ when you really care about speed, using premultiplied pixels for both in
10868+ and out (STBIR_RGBA_PM, etc) much faster than both of these premultiplied
10869+ options.
10870+
10871+ PIXEL LAYOUT CONVERSION
10872+ The resizer can convert from some pixel layouts to others. When using the
10873+ stbir_set_pixel_layouts(), you can, for example, specify STBIR_RGBA
10874+ on input, and STBIR_ARGB on output, and it will re-organize the channels
10875+ during the resize. Currently, you can only convert between two pixel
10876+ layouts with the same number of channels.
10877+
10878+ DETERMINISM
10879+ We commit to being deterministic (from x64 to ARM to scalar to SIMD, etc).
10880+ This requires compiling with fast-math off (using at least /fp:precise).
10881+ Also, you must turn off fp-contracting (which turns mult+adds into fmas)!
10882+ We attempt to do this with pragmas, but with Clang, you usually want to add
10883+ -ffp-contract=off to the command line as well.
10884+
10885+ For 32-bit x86, you must use SSE and SSE2 codegen for determinism. That is,
10886+ if the scalar x87 unit gets used at all, we immediately lose determinism.
10887+ On Microsoft Visual Studio 2008 and earlier, from what we can tell there is
10888+ no way to be deterministic in 32-bit x86 (some x87 always leaks in, even
10889+ with fp:strict). On 32-bit x86 GCC, determinism requires both -msse2 and
10890+ -fpmath=sse.
10891+
10892+ Note that we will not be deterministic with float data containing NaNs -
10893+ the NaNs will propagate differently on different SIMD and platforms.
10894+
10895+ If you turn on STBIR_USE_FMA, then we will be deterministic with other
10896+ fma targets, but we will differ from non-fma targets (this is unavoidable,
10897+ because a fma isn't simply an add with a mult - it also introduces a
10898+ rounding difference compared to non-fma instruction sequences.
10899+
10900+ FLOAT PIXEL FORMAT RANGE
10901+ Any range of values can be used for the non-alpha float data that you pass
10902+ in (0 to 1, -1 to 1, whatever). However, if you are inputting float values
10903+ but *outputting* bytes or shorts, you must use a range of 0 to 1 so that we
10904+ scale back properly. The alpha channel must also be 0 to 1 for any format
10905+ that does premultiplication prior to resizing.
10906+
10907+ Note also that with float output, using filters with negative lobes, the
10908+ output filtered values might go slightly out of range. You can define
10909+ STBIR_FLOAT_LOW_CLAMP and/or STBIR_FLOAT_HIGH_CLAMP to specify the range
10910+ to clamp to on output, if that's important.
10911+
10912+ MAX/MIN SCALE FACTORS
10913+ Images can be up to 2^30 pixels wide or high (dimensions are ints, and
10914+ the internal pointer math is done in size_t).
10915+
10916+ The positions of the filter taps are calculated in stbir_real (double by
10917+ default), which is accurate all the way to that limit. When the scale
10918+ works out to an exact fraction (which it does for most whole image
10919+ resizes), the positions are calculated with 64-bit integer math and are
10920+ exact at any size.
10921+
10922+ If you are on a part without hardware doubles, you can define stbir_real
10923+ to float before you #include. Positions are then accurate to about 2^20
10924+ pixels and are only off by half a pixel at 2^24, so for extreme resizes
10925+ on those parts, do it in multiple stages (using float intermediate
10926+ buffers).
10927+
10928+ HUGE IMAGES AND TILING
10929+ Image strides are 64-bit, so the buffers themselves can be as large as
10930+ memory allows; widths and heights are limited to 2^30 (ints). To resize
10931+ an image larger than that in either direction, resize it in tiles: for
10932+ each output tile, point the resize at a piece of the input (the 64-bit
10933+ strides reach anywhere in the buffer) with the output tile's size, and
10934+ set the input region the tile maps to, in the piece's own coordinates,
10935+ with stbir_set_input_subrect. Two things keep the tiles seamless:
10936+
10937+ - the piece must extend past the region by the filter's reach on every
10938+ side that isn't a real image edge (the filter's support in input
10939+ pixels: support x input / output when downsampling, support when
10940+ upsampling, plus a pixel of slack), because the edge modes act at
10941+ the piece's edges - a tap that runs off the piece is clamped or
10942+ wrapped there, as if the image ended. WRAP is the awkward one: a
10943+ tile at a real edge needs the pixels from the opposite edge of the
10944+ whole image, which a piece of the buffer can't reach, so you would
10945+ have to copy them into place next to the piece - it is simpler to
10946+ use another edge mode when tiling.
10947+
10948+ - the region must be exactly the tile's share of the whole mapping
10949+ (tile size x whole input / whole output, at the tile's offset), so
10950+ every tile has the whole resize's scale and phase - a region drawn
10951+ from rounded numbers shifts that tile's samples against its
10952+ neighbors.
10953+
10954+ A region that reaches past the source is clipped to it (the output
10955+ pixels that would map outside are not written).
10956+
10957+ FLIPPED IMAGES
10958+ Stride is just the delta from one scanline to the next. This means you can
10959+ use a negative stride to handle inverted images (point to the final
10960+ scanline and use a negative stride). You can invert the input or output,
10961+ using negative strides.
10962+
10963+ DEFAULT FILTERS
10964+ For functions which don't provide explicit control over what filters to
10965+ use, you can change the compile-time defaults with:
10966+
10967+ #define STBIR_DEFAULT_FILTER_UPSAMPLE STBIR_FILTER_something
10968+ #define STBIR_DEFAULT_FILTER_DOWNSAMPLE STBIR_FILTER_something
10969+
10970+ See stbir_filter in the header-file section for the list of filters.
10971+
10972+ NEW FILTERS
10973+ A number of 1D filter kernels are supplied. For a list of supported
10974+ filters, see the stbir_filter enum. You can install your own filters by
10975+ using the stbir_set_filter_callbacks_new function.
10976+
10977+ PROGRESS
10978+ For interactive use with slow resize operations, you can use the
10979+ scanline callbacks in the extended API. It would have to be a *very* large
10980+ image resample to need progress though - we're very fast.
10981+
10982+ ASSERT
10983+ Define STBIR_ASSERT(boolval) to override assert() and not use assert.h
10984+
10985+ CACHE SIZE
10986+ A resize is run in column slices sized so that its working memory fits
10987+ the L2 cache. By default, the L2 size is read with cpuid on x86 (and a L2
10988+ of a megabyte or more is treated as 512 KB, to keep the internal memory
10989+ down), and taken as 256 KB on other cpus. If that isn't right for you,
10990+ set stbir_l2_cache_size_override (bytes) before resizing, and it is used
10991+ as is; or define STBIR_ASK_WINDOWS_FOR_L2_SIZE, STBIR_ASK_LINUX_FOR_L2_SIZE
10992+ or STBIR_ASK_MACOS_FOR_L2_SIZE before the implementation include to ask
10993+ the OS instead of cpuid (see the comment above them for the headers each
10994+ one needs). Usually the defaults will just work fine.
10995+
10996+ THREADING
10997+ To resize on several threads, call stbir_build_samplers_with_splits on
10998+ one thread with the number of threads you'd like, then call
10999+ stbir_resize_extended_split on each thread with its index (and a count
11000+ of 1), and stbir_free_samplers when they're all done. The build returns
11001+ the number of splits to actually use: it is limited to how many ways
11002+ the resize can be split, and then to the count the build estimates
11003+ will still speed things up (a small image or very narrow rows don't
11004+ get faster with more threads, they get slower). The estimate's constants
11005+ are statics in the implementation that you can override if you need to.
11006+
11007+ FUTURE TODOS
11008+ * Add pixel layout conversions for sensible different channel counts
11009+ (maybe, 1->3/4, 3->4, 4->1, 3->1).
11010+ * Convert the reincludes to macros when we know they aren't changing.
11011+ * Most of our code is internally function pointers, should we compile
11012+ all the SIMD stuff always and dynamically dispatch?
11013+
11014+ CONTRIBUTORS
11015+ Jeff Roberts: 2.0 implementation, optimizations, SIMD
11016+ Martins Mozeiko: NEON simd, WASM simd, clang and GCC whisperer
11017+ Fabian Giesen: half float and srgb converters
11018+ Sean Barrett: API design, optimizations
11019+ Jorge L Rodriguez: Original 1.0 implementation
11020+ Aras Pranckevicius: bugfixes
11021+ Nathan Reed: warning fixes for 1.0
11022+
11023+ REVISIONS
11024+ 2.50 (2026-10-07) Big performance release, especially for large and/or
11025+ threaded resizes! New stuff:
11026+ - Resizes now run in column slices sized to the L2 cache
11027+ (auto-detected, or set stbir_l2_cache_size_override),
11028+ which is 1.6-3x faster on large resizes (more when
11029+ threaded) and uses far less temp memory.
11030+ - Coefficient sets for rational scales are now shared,
11031+ using much less memory for these resizes (and faster).
11032+ - Rational downsamples now get faster dedicated kernels
11033+ (20-30% faster).
11034+ - Added horizontal gather kernels up to 17 taps, so
11035+ 4-tap filters stay on the fast gather path down to
11036+ 4:1 (20-30% faster than scatter path). Useful in mip
11037+ chain generation loops.
11038+ - For each resize, we adjust the offset of the internal
11039+ buffers to avoid 4K aliasing with the input buffer
11040+ (avoids random 40% slowdowns).
11041+ - New model for estimating vertical or horizontal first
11042+ mode. Much better at predicting best direction (lots
11043+ of resizes faster 30%-40% with this change).
11044+ - Added a model of thread addition improvement - if you
11045+ ask for 32 threads for example, the model will only use
11046+ the thread count that will improve performance (some
11047+ users were estimating this themselves - this should
11048+ be more accurate).
11049+ - 1 channel downsizes 30-40% faster (SIMD across samples).
11050+ - Added some new AVX kernels and optimizations.
11051+ - Added AVX-512 support - useful mostly just for Zen 5
11052+ and recent Intel cores and large resamples.
11053+ - Bunch of cleanup and exactness in coefficient building,
11054+ along with new higher precision filter callbacks - this
11055+ moves our image resize limits up to 2^30.
11056+ - Added several new filters - more Mitchell variants,
11057+ some Lanczos, and the Muratori filter.
11058+ - Point sampling now actually does point sampling when
11059+ downsampling (previously only upsamples were point
11060+ sampled, down was averaged).
11061+ - Added 64-bit strides so that you can resize over the
11062+ 2^30 with manual tiling (calling resize multiple
11063+ times over the 64-bit area).
11064+ - Windows ARM64EC support (thanks navvyswethgraphics)
11065+ - Lots of little bugfixes in unusual corner cases.
11066+ NOTE THAT 2.50 IS NOT deterministic with prior versions
11067+ (among other improvements: coefficients calculated more
11068+ precisely, new v-first vs h-first decisions, etc).
11069+
11070+ 2.18 (2026-03-25) fixed coefficient calculation when skipping a coefficient off
11071+ the left side of the window, added non-aligned access safe
11072+ memcpy mode for scalar path, fixed various typos, and fixed
11073+ define error in the float clamp output mode.
11074+ 2.17 (2025-10-25) silly format bug in easy-to-use APIs.
11075+ 2.16 (2025-10-21) fixed the easy-to-use APIs to allow inverted bitmaps (negative
11076+ strides), fix vertical filter kernel callback, fix threaded
11077+ gather buffer priming (and assert).
11078+ (thanks adipose, TainZerL, and Harrison Green)
11079+ 2.15 (2025-07-17) fixed an assert in debug mode when using floats with input
11080+ callbacks, work around GCC warning when adding to null ptr
11081+ (thanks Johannes Spohr and Pyry Kovanen).
11082+ 2.14 (2025-05-09) fixed a bug using downsampling gather horizontal first, and
11083+ scatter with vertical first.
11084+ 2.13 (2025-02-27) fixed a bug when using input callbacks, turned off simd for
11085+ tiny-c, fixed some variables that should have been static,
11086+ fixes a bug when calculating temp memory with resizes that
11087+ exceed 2GB of temp memory (very large resizes).
11088+ 2.12 (2024-10-18) fix incorrect use of user_data with STBIR_FREE
11089+ 2.11 (2024-09-08) fix harmless asan warnings in 2-channel and 3-channel mode
11090+ with AVX-2, fix some weird scaling edge conditions with
11091+ point sample mode.
11092+ 2.10 (2024-07-27) fix the defines GCC and mingw for loop unroll control,
11093+ fix MSVC 32-bit arm half float routines.
11094+ 2.09 (2024-06-19) fix the defines for 32-bit ARM GCC builds (was selecting
11095+ hardware half floats).
11096+ 2.08 (2024-06-10) fix for RGB->BGR three channel flips and add SIMD (thanks
11097+ to Ryan Salsbury), fix for sub-rect resizes, use the
11098+ pragmas to control unrolling when they are available.
11099+ 2.07 (2024-05-24) fix for slow final split during threaded conversions of very
11100+ wide scanlines when downsampling (caused by extra input
11101+ converting), fix for wide scanline resamples with many
11102+ splits (int overflow), fix GCC warning.
11103+ 2.06 (2024-02-10) fix for identical width/height 3x or more down-scaling
11104+ undersampling a single row on rare resize ratios (about 1%).
11105+ 2.05 (2024-02-07) fix for 2 pixel to 1 pixel resizes with wrap (thanks Aras),
11106+ fix for output callback (thanks Julien Koenen).
11107+ 2.04 (2023-11-17) fix for rare AVX bug, shadowed symbol (thanks Nikola Smiljanic).
11108+ 2.03 (2023-11-01) ASAN and TSAN warnings fixed, minor tweaks.
11109+ 2.00 (2023-10-10) mostly new source: new api, optimizations, simd, vertical-first, etc
11110+ 2x-5x faster without simd, 4x-12x faster with simd,
11111+ in some cases, 20x to 40x faster esp resizing large to very small.
11112+ 0.96 (2019-03-04) fixed warnings
11113+ 0.95 (2017-07-23) fixed warnings
11114+ 0.94 (2017-03-18) fixed warnings
11115+ 0.93 (2017-03-03) fixed bug with certain combinations of heights
11116+ 0.92 (2017-01-02) fix integer overflow on large (>2GB) images
11117+ 0.91 (2016-04-02) fix warnings; fix handling of subpixel regions
11118+ 0.90 (2014-09-17) first released version
11119+
11120+ LICENSE
11121+ See end of file for license information.
11122+*/
11123+
11124+#if !defined(STB_IMAGE_RESIZE_DO_HORIZONTALS) && !defined(STB_IMAGE_RESIZE_DO_VERTICALS) && !defined(STB_IMAGE_RESIZE_DO_CODERS) // for internal re-includes
11125+
11126+#ifndef STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
11127+#define STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
11128+
11129+#include <stddef.h>
11130+#ifdef _MSC_VER
11131+typedef unsigned char stbir_uint8;
11132+typedef unsigned short stbir_uint16;
11133+typedef unsigned int stbir_uint32;
11134+typedef unsigned __int64 stbir_uint64;
11135+typedef __int64 stbir_int64;
11136+#else
11137+#include <stdint.h>
11138+typedef uint8_t stbir_uint8;
11139+typedef uint16_t stbir_uint16;
11140+typedef uint32_t stbir_uint32;
11141+typedef uint64_t stbir_uint64;
11142+typedef int64_t stbir_int64;
11143+#endif
11144+
11145+// we use this type for high precision floating point work - it's normally just a double
11146+// but you can define it to float, if you don't have double support (some old embedded
11147+// platforms).
11148+#ifndef stbir_real
11149+#define stbir_real double
11150+#endif
11151+
11152+#ifndef STBIRDEF
11153+#ifdef STB_IMAGE_RESIZE_STATIC
11154+#define STBIRDEF static
11155+#else
11156+#ifdef __cplusplus
11157+#define STBIRDEF extern "C"
11158+#else
11159+#define STBIRDEF extern
11160+#endif
11161+#endif
11162+#endif
11163+
11164+//////////////////////////////////////////////////////////////////////////////
11165+//// start "header file" ///////////////////////////////////////////////////
11166+//
11167+// Easy-to-use API:
11168+//
11169+// * stride is the offset between successive rows of image data
11170+// in memory, in bytes (64-bit, so a buffer can be larger than 2 GB).
11171+// specify 0 for packed continuously in memory
11172+// * colorspace is linear or sRGB as specified by function name
11173+// * Uses the default filters
11174+// * Uses edge mode clamped
11175+// * returned result is 1 for success or 0 in case of an error.
11176+
11177+
11178+// stbir_pixel_layout specifies:
11179+// number of channels
11180+// order of channels
11181+// whether color is premultiplied by alpha
11182+// for back compatibility, you can cast the old channel count to an stbir_pixel_layout
11183+typedef enum
11184+{
11185+ STBIR_1CHANNEL = 1,
11186+ STBIR_2CHANNEL = 2,
11187+ STBIR_RGB = 3, // 3-chan, with order specified (for channel flipping)
11188+ STBIR_BGR = 0, // 3-chan, with order specified (for channel flipping)
11189+ STBIR_4CHANNEL = 5,
11190+
11191+ STBIR_RGBA = 4, // alpha formats, where alpha is NOT premultiplied into color channels
11192+ STBIR_BGRA = 6,
11193+ STBIR_ARGB = 7,
11194+ STBIR_ABGR = 8,
11195+ STBIR_RA = 9,
11196+ STBIR_AR = 10,
11197+
11198+ STBIR_RGBA_PM = 11, // alpha formats, where alpha is premultiplied into color channels
11199+ STBIR_BGRA_PM = 12,
11200+ STBIR_ARGB_PM = 13,
11201+ STBIR_ABGR_PM = 14,
11202+ STBIR_RA_PM = 15,
11203+ STBIR_AR_PM = 16,
11204+
11205+ STBIR_RGBA_NO_AW = 11, // alpha formats, where NO alpha weighting is applied at all!
11206+ STBIR_BGRA_NO_AW = 12, // these are just synonyms for the _PM flags (which also do
11207+ STBIR_ARGB_NO_AW = 13, // no alpha weighting). These names just make it more clear
11208+ STBIR_ABGR_NO_AW = 14, // for some folks).
11209+ STBIR_RA_NO_AW = 15,
11210+ STBIR_AR_NO_AW = 16,
11211+
11212+} stbir_pixel_layout;
11213+
11214+//===============================================================
11215+// Simple-complexity API
11216+//
11217+// If output_pixels is NULL (0), then we will allocate the buffer and return it to you.
11218+//--------------------------------
11219+
11220+STBIRDEF unsigned char * stbir_resize_uint8_srgb( const unsigned char *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
11221+ unsigned char *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
11222+ stbir_pixel_layout pixel_type );
11223+
11224+STBIRDEF unsigned char * stbir_resize_uint8_linear( const unsigned char *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
11225+ unsigned char *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
11226+ stbir_pixel_layout pixel_type );
11227+
11228+STBIRDEF float * stbir_resize_float_linear( const float *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
11229+ float *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
11230+ stbir_pixel_layout pixel_type );
11231+//===============================================================
11232+
11233+//===============================================================
11234+// Medium-complexity API
11235+//
11236+// This extends the easy-to-use API as follows:
11237+//
11238+// * Can specify the datatype - U8, U8_SRGB, U16, FLOAT, HALF_FLOAT
11239+// * Edge wrap can selected explicitly
11240+// * Filter can be selected explicitly
11241+//--------------------------------
11242+
11243+typedef enum
11244+{
11245+ STBIR_EDGE_CLAMP = 0,
11246+ STBIR_EDGE_REFLECT = 1,
11247+ STBIR_EDGE_WRAP = 2, // this edge mode is slower and uses more memory
11248+ STBIR_EDGE_ZERO = 3,
11249+} stbir_edge;
11250+
11251+typedef enum
11252+{
11253+ STBIR_FILTER_DEFAULT = 0, // use same filter type that easy-to-use API chooses
11254+ STBIR_FILTER_BOX = 1, // A trapezoid w/1-pixel wide ramps, same result as box for integer scale ratios
11255+ STBIR_FILTER_TRIANGLE = 2, // On upsampling, produces same results as bilinear texture filtering
11256+ STBIR_FILTER_CUBICBSPLINE = 3, // The cubic b-spline (aka Mitchell-Netrevalli with B=1,C=0), gaussian-esque
11257+ STBIR_FILTER_CATMULLROM = 4, // An interpolating cubic spline (the default when upsampling)
11258+ STBIR_FILTER_MITCHELL = 5, // Mitchell-Netrevalli filter with B=1/3, C=1/3 (the default when downsampling)
11259+ STBIR_FILTER_POINT_SAMPLE = 6, // Simple point sampling
11260+ STBIR_FILTER_MITCHELL_QUARTER = 7, // Mitchell-Netrevalli with B=1/4, C=3/8 (Robidoux), a little sharper than Mitchell
11261+ STBIR_FILTER_MITCHELL_SIXTH = 8, // Mitchell-Netrevalli with B=1/6, C=5/12, sharper still
11262+ STBIR_FILTER_MITCHELL_NEG_SIXTH = 9, // Mitchell-Netrevalli with B=-1/6, C=7/12, over-sharpening
11263+ STBIR_FILTER_MITCHELL_NEG_THIRD = 10, // Mitchell-Netrevalli with B=-1/3, C=2/3, more over-sharpening
11264+ STBIR_FILTER_LANCZOS4 = 11, // Lanczos with 2 lobes (4 taps at 1:1), sharp, some ringing on hard edges
11265+ STBIR_FILTER_LANCZOS5 = 12, // Lanczos with 2.5 lobes (5 taps at 1:1), very sharp, often ringy
11266+ STBIR_FILTER_MURATORI6 = 13, // Casey Muratori's Bink 2 kernel (1 -4 19 19 -4 1)/32: sharper than Catmull-Rom, stable under repeated half pixel shifts when using 8-bit pixels
11267+ STBIR_FILTER_OTHER = 14, // User callback specified (keep last - the built-ins are everything below it)
11268+} stbir_filter;
11269+
11270+typedef enum
11271+{
11272+ STBIR_TYPE_UINT8 = 0,
11273+ STBIR_TYPE_UINT8_SRGB = 1,
11274+ STBIR_TYPE_UINT8_SRGB_ALPHA = 2, // alpha channel, when present, should also be SRGB (this is very unusual)
11275+ STBIR_TYPE_UINT16 = 3,
11276+ STBIR_TYPE_FLOAT = 4,
11277+ STBIR_TYPE_HALF_FLOAT = 5
11278+} stbir_datatype;
11279+
11280+// medium api
11281+STBIRDEF void * stbir_resize( const void *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
11282+ void *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
11283+ stbir_pixel_layout pixel_layout, stbir_datatype data_type,
11284+ stbir_edge edge, stbir_filter filter );
11285+//===============================================================
11286+
11287+
11288+//===============================================================
11289+// Extended-complexity API
11290+//
11291+// This API exposes all resize functionality.
11292+//
11293+// * Separate filter types for each axis
11294+// * Separate edge modes for each axis
11295+// * Separate input and output data types
11296+// * Can specify regions with subpixel correctness
11297+// * Can specify alpha flags
11298+// * Can specify a memory callback
11299+// * Can specify a callback data type for pixel input and output
11300+// * Can be threaded for a single resize
11301+// * Can be used to resize many frames without recalculating the sampler info
11302+//
11303+// Use this API as follows:
11304+// 1) Call the stbir_resize_init function on a local STBIR_RESIZE structure
11305+// 2) Call any of the stbir_set functions
11306+// 3) Optionally call stbir_build_samplers() if you are going to resample multiple times
11307+// with the same input and output dimensions (like resizing video frames)
11308+// 4) Resample by calling stbir_resize_extended().
11309+// 5) Call stbir_free_samplers() if you called stbir_build_samplers()
11310+//--------------------------------
11311+
11312+
11313+// Types:
11314+
11315+// INPUT CALLBACK: this callback is used for input scanlines
11316+typedef void const * stbir_input_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context );
11317+
11318+// OUTPUT CALLBACK: this callback is used for output scanlines (or pieces of scanlines - see below)
11319+typedef void stbir_output_callback( void const * output_ptr, int num_pixels, int x, int y, void * context );
11320+
11321+// callbacks for user installed filters (there are old and new versions, please use the new versions from now on)
11322+typedef float stbir__kernel_callback( float x, float scale, void * user_data );
11323+typedef float stbir__support_callback( float scale, void * user_data );
11324+typedef stbir_real stbir__kernel_callback_new( stbir_real x, stbir_real scale, void * user_data );
11325+typedef stbir_real stbir__support_callback_new( stbir_real scale, void * user_data );
11326+typedef stbir_real stbir__radius_callback_new( stbir_real scale, stbir_real inv_scale, int is_upsample, void * user_data );
11327+
11328+// internal structure with precomputed scaling
11329+typedef struct stbir__info stbir__info;
11330+
11331+typedef struct STBIR_RESIZE // use the stbir_resize_init and stbir_override functions to set these values for future compatibility
11332+{
11333+ void * user_data;
11334+ void const * input_pixels;
11335+ int input_w, input_h;
11336+ stbir_real input_s0, input_t0, input_s1, input_t1;
11337+ int input_s_n0, input_s_n1, input_s_d; // the input region as fractions, when it was set in pixels: s0 = n0/d, s1 = n1/d (d is 0 when set as reals)
11338+ int input_t_n0, input_t_n1, input_t_d;
11339+ stbir_input_callback * input_cb;
11340+ void * output_pixels;
11341+ int output_w, output_h;
11342+ int output_subx, output_suby, output_subw, output_subh;
11343+ stbir_output_callback * output_cb;
11344+ stbir_int64 input_stride_in_bytes;
11345+ stbir_int64 output_stride_in_bytes;
11346+ int splits;
11347+ int fast_alpha;
11348+ int needs_rebuild;
11349+ int called_alloc;
11350+ stbir_pixel_layout input_pixel_layout_public;
11351+ stbir_pixel_layout output_pixel_layout_public;
11352+ stbir_datatype input_data_type;
11353+ stbir_datatype output_data_type;
11354+ stbir_filter horizontal_filter, vertical_filter;
11355+ stbir_edge horizontal_edge, vertical_edge;
11356+ stbir__kernel_callback * horizontal_filter_kernel; stbir__support_callback * horizontal_filter_support; // old callback form
11357+ stbir__kernel_callback * vertical_filter_kernel; stbir__support_callback * vertical_filter_support;
11358+ stbir__kernel_callback_new * horizontal_filter_kernel_new; stbir__support_callback_new * horizontal_filter_support_new; stbir__radius_callback_new * horizontal_filter_radius;
11359+ stbir__kernel_callback_new * vertical_filter_kernel_new; stbir__support_callback_new * vertical_filter_support_new; stbir__radius_callback_new * vertical_filter_radius;
11360+ stbir__info * samplers;
11361+} STBIR_RESIZE;
11362+
11363+// extended complexity api
11364+
11365+
11366+// First off, you must ALWAYS call stbir_resize_init on your resize structure before any of the other calls!
11367+STBIRDEF void stbir_resize_init( STBIR_RESIZE * resize,
11368+ const void *input_pixels, int input_w, int input_h, stbir_int64 input_stride_in_bytes, // stride can be zero
11369+ void *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes, // stride can be zero
11370+ stbir_pixel_layout pixel_layout, stbir_datatype data_type );
11371+
11372+//===============================================================
11373+// You can update these parameters any time after resize_init and there is no cost
11374+//--------------------------------
11375+
11376+STBIRDEF void stbir_set_datatypes( STBIR_RESIZE * resize, stbir_datatype input_type, stbir_datatype output_type );
11377+STBIRDEF void stbir_set_pixel_callbacks( STBIR_RESIZE * resize, stbir_input_callback * input_cb, stbir_output_callback * output_cb ); // no callbacks by default
11378+STBIRDEF void stbir_set_user_data( STBIR_RESIZE * resize, void * user_data ); // pass back STBIR_RESIZE* by default
11379+STBIRDEF void stbir_set_buffer_ptrs( STBIR_RESIZE * resize, const void * input_pixels, stbir_int64 input_stride_in_bytes, void * output_pixels, stbir_int64 output_stride_in_bytes );
11380+
11381+//===============================================================
11382+
11383+
11384+//===============================================================
11385+// If you call any of these functions, you will trigger a sampler rebuild!
11386+//--------------------------------
11387+
11388+STBIRDEF int stbir_set_pixel_layouts( STBIR_RESIZE * resize, stbir_pixel_layout input_pixel_layout, stbir_pixel_layout output_pixel_layout ); // sets new buffer layouts
11389+STBIRDEF int stbir_set_edgemodes( STBIR_RESIZE * resize, stbir_edge horizontal_edge, stbir_edge vertical_edge ); // CLAMP by default
11390+
11391+STBIRDEF int stbir_set_filters( STBIR_RESIZE * resize, stbir_filter horizontal_filter, stbir_filter vertical_filter ); // STBIR_DEFAULT_FILTER_UPSAMPLE/DOWNSAMPLE by default
11392+
11393+// Your own filters: per axis, a kernel callback and one of two ways to give its extent (a support or a radius - both
11394+// on one axis returns 0; all null keeps the built-in filter). All in stbir_real (double unless user overridden to float).
11395+// the resizer places the taps in stbir_real and rounds each weight to float once, into the coefficient table.
11396+//
11397+// The kernel returns the weight at x, in the filter's own pixels (output pixels when downsampling, input pixels when
11398+// upsampling), centered on zero. It is also passed the ratio <= 1 (output / input downsampling, input / output
11399+// upsampling), for kernels like the box whose shape depends on it.
11400+//
11401+// The support returns the filter's radius in those same filter pixels, and is passed the same ratio. For most kernels
11402+// this is a constant (2 for the cubics, 1 for the triangle); the resizer converts it to input pixels itself.
11403+//
11404+// The radius is the alternative for a kernel whose extent depends on the scale: it returns the radius in INPUT pixels,
11405+// from scale, inv_scale and the resizer's own up / down decision (is_upsample, 1:1 included), so an exact ratio gets
11406+// an exact tap count (the built-in box is 0.5 * inv_scale + 0.5).
11407+//
11408+// static stbir_real my_kernel( stbir_real x, stbir_real scale, void * user ) { ... }
11409+// static stbir_real my_support( stbir_real scale, void * user ) { return 2.0; }
11410+// stbir_set_filter_callbacks_new( &resize, my_kernel, my_support, 0, my_kernel, my_support, 0 );
11411+STBIRDEF int stbir_set_filter_callbacks_new( STBIR_RESIZE * resize, stbir__kernel_callback_new * horizontal_filter, stbir__support_callback_new * horizontal_support, stbir__radius_callback_new * horizontal_radius, stbir__kernel_callback_new * vertical_filter, stbir__support_callback_new * vertical_support, stbir__radius_callback_new * vertical_radius );
11412+
11413+// Old version of the filter callbacks (for backwards compatibility) - please use stbir_set_filter_callbacks_new for new code.
11414+STBIRDEF int stbir_set_filter_callbacks( STBIR_RESIZE * resize, stbir__kernel_callback * horizontal_filter, stbir__support_callback * horizontal_support, stbir__kernel_callback * vertical_filter, stbir__support_callback * vertical_support );
11415+
11416+STBIRDEF int stbir_set_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ); // sets both sub-regions (full regions by default)
11417+STBIRDEF int stbir_set_input_subrect( STBIR_RESIZE * resize, stbir_real s0, stbir_real t0, stbir_real s1, stbir_real t1 ); // sets input sub-region (full region by default). a region past the image is clipped to it: the output pixels mapping outside aren't written
11418+STBIRDEF int stbir_set_output_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ); // sets output sub-region (full region by default)
11419+
11420+// when inputting AND outputting non-premultiplied alpha pixels, we use a slower but higher quality technique
11421+// that fills the zero alpha pixel's RGB values with something plausible. If you don't care about areas of
11422+// zero alpha, you can call this function to get about a 25% speed improvement for STBIR_RGBA to STBIR_RGBA
11423+// types of resizes.
11424+STBIRDEF int stbir_set_non_pm_alpha_speed_over_quality( STBIR_RESIZE * resize, int non_pma_alpha_speed_over_quality );
11425+//===============================================================
11426+
11427+
11428+//===============================================================
11429+// You can call build_samplers to prebuild all the internal data we need to resample.
11430+// Then, if you call resize_extended many times with the same resize, you only pay the
11431+// cost once.
11432+// If you do call build_samplers, you MUST call free_samplers eventually.
11433+//--------------------------------
11434+
11435+// This builds the samplers and does one allocation
11436+STBIRDEF int stbir_build_samplers( STBIR_RESIZE * resize );
11437+
11438+// You MUST call this, if you call stbir_build_samplers or stbir_build_samplers_with_splits
11439+STBIRDEF void stbir_free_samplers( STBIR_RESIZE * resize );
11440+//===============================================================
11441+
11442+
11443+// And this is the main function to perform the resize synchronously on one thread.
11444+STBIRDEF int stbir_resize_extended( STBIR_RESIZE * resize );
11445+
11446+
11447+//===============================================================
11448+// Use these functions for multithreading.
11449+// 1) You call stbir_build_samplers_with_splits first on the main thread
11450+// 2) Then stbir_resize_extended_split on each thread
11451+// 3) stbir_free_samplers when done on the main thread
11452+//--------------------------------
11453+
11454+// This will build samplers for threading.
11455+// You can pass in the number of threads you'd like to use (try_splits).
11456+// It returns the number of splits (threads) that you can call it with.
11457+// It might be less: when the image can't be split up that many ways, or when
11458+// the build's estimate says more threads wouldn't help.
11459+
11460+STBIRDEF int stbir_build_samplers_with_splits( STBIR_RESIZE * resize, int try_splits );
11461+
11462+// This function does a split of the resizing (you call this fuction for each
11463+// split, on multiple threads). A split is a piece of the output resize pixel space.
11464+
11465+// Note that you MUST call stbir_build_samplers_with_splits before stbir_resize_extended_split!
11466+
11467+// Usually, you will always call stbir_resize_extended_split with split_start as the thread_index
11468+// and "1" for the split_count.
11469+// But, if you have a weird situation where you MIGHT want 8 threads, but sometimes
11470+// only 4 threads, you can use 0,2,4,6 for the split_start's and use "2" for the
11471+// split_count each time to turn in into a 4 thread resize. (This is unusual).
11472+
11473+STBIRDEF int stbir_resize_extended_split( STBIR_RESIZE * resize, int split_start, int split_count );
11474+//===============================================================
11475+
11476+
11477+//===============================================================
11478+// Pixel Callbacks info:
11479+//--------------------------------
11480+
11481+// The input callback is super flexible - it calls you with the input address
11482+// of the start of the input row (based on the stride and base pointer), and the
11483+// x position and pixel count that it needs from that row. It gives you an
11484+// optional_output pointer that you can fill, or you can just return your own
11485+// pointer into your own data.
11486+//
11487+// You can also do conversion from non-supported data types if necessary - in
11488+// this case, you ignore the input_ptr and just use the x and y parameters to
11489+// calculate your own input_ptr based on the size of each non-supported pixel.
11490+// (Something like the third example below.)
11491+//
11492+// You can also install just an input or just an output callback by setting the
11493+// callback that you don't want to zero.
11494+//
11495+// First example, progress: (getting a callback that you can monitor the progress):
11496+// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
11497+// {
11498+// percentage_done = y / input_height;
11499+// return ( (char*) input_ptr ) + x * pixel_width_in_bytes; // use buffer from call (input_ptr is the row start)
11500+// }
11501+//
11502+// Next example, copying: (copy from some other buffer or stream):
11503+// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
11504+// {
11505+// CopyOrStreamData( optional_output, other_data_src, num_pixels * pixel_width_in_bytes );
11506+// return optional_output; // return the optional buffer that we filled
11507+// }
11508+//
11509+// Third example, input another buffer without copying: (zero-copy from other buffer):
11510+// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
11511+// {
11512+// void * pixels = ( (char*) other_image_base ) + ( y * other_image_stride ) + ( x * other_pixel_width_in_bytes );
11513+// return pixels; // return pointer to your data without copying
11514+// }
11515+//
11516+//
11517+// The output callback is considerably simpler - it just calls you so that you can dump
11518+// out the output pixels. You can also convert to other output types here if you want.
11519+//
11520+// Note that the resizer can process the output in column slices (to stay in cache), so
11521+// you are called with pieces of scanlines: num_pixels pixels starting at output pixel x
11522+// on output row y. x and y are always in the full output image's pixel space (so with an
11523+// output sub-rect, they include the sub-rect offset). The pieces of a scanline are not
11524+// delivered in order, so if you want whole scanlines, assemble them yourself with x and y.
11525+//
11526+// Simple example:
11527+// void my_output( void const * output_ptr, int num_pixels, int x, int y, void * context )
11528+// {
11529+// memcpy( ( (char*) my_output_image ) + y * my_output_stride + x * pixel_width_in_bytes, output_ptr, num_pixels * pixel_width_in_bytes );
11530+// }
11531+//===============================================================
11532+
11533+
11534+
11535+//===============================================================
11536+// optional built-in profiling API
11537+//--------------------------------
11538+
11539+#ifdef STBIR_PROFILE
11540+
11541+typedef struct STBIR_PROFILE_INFO
11542+{
11543+ stbir_uint64 total_clocks;
11544+
11545+ // how many clocks spent (of total_clocks) in the various resize routines, along with a string description
11546+ // there are "resize_count" number of zones
11547+ stbir_uint64 clocks[ 8 ];
11548+ char const ** descriptions;
11549+
11550+ // count of clocks and descriptions
11551+ stbir_uint32 count;
11552+} STBIR_PROFILE_INFO;
11553+
11554+// use after calling stbir_resize_extended (or stbir_build_samplers or stbir_build_samplers_with_splits)
11555+STBIRDEF void stbir_resize_build_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize );
11556+
11557+// use after calling stbir_resize_extended
11558+STBIRDEF void stbir_resize_extended_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize );
11559+
11560+// use after calling stbir_resize_extended_split
11561+STBIRDEF void stbir_resize_split_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize, int split_start, int split_num );
11562+
11563+//===============================================================
11564+
11565+#endif
11566+
11567+
11568+//// end header file /////////////////////////////////////////////////////
11569+#endif // STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
11570+
11571+#if defined(STB_IMAGE_RESIZE_IMPLEMENTATION) || defined(STB_IMAGE_RESIZE2_IMPLEMENTATION)
11572+
11573+#ifndef STBIR_ASSERT
11574+#include <assert.h>
11575+#define STBIR_ASSERT(x) assert(x)
11576+#endif
11577+
11578+#ifndef STBIR_MALLOC
11579+#include <stdlib.h>
11580+#define STBIR_MALLOC(size,user_data) ((void)(user_data), malloc(size))
11581+#define STBIR_FREE(ptr,user_data) ((void)(user_data), free(ptr))
11582+// (we used the comma operator to evaluate user_data, to avoid "unused parameter" warnings)
11583+#endif
11584+
11585+#ifdef _MSC_VER
11586+
11587+#define stbir__inline __forceinline
11588+
11589+#else
11590+
11591+#define stbir__inline __inline__
11592+
11593+// Clang address sanitizer
11594+#if defined(__has_feature)
11595+ #if __has_feature(address_sanitizer) || __has_feature(memory_sanitizer)
11596+ #ifndef STBIR__SEPARATE_ALLOCATIONS
11597+ #define STBIR__SEPARATE_ALLOCATIONS
11598+ #endif
11599+ #endif
11600+#endif
11601+
11602+#endif
11603+
11604+// GCC and MSVC
11605+#if defined(__SANITIZE_ADDRESS__)
11606+ #ifndef STBIR__SEPARATE_ALLOCATIONS
11607+ #define STBIR__SEPARATE_ALLOCATIONS
11608+ #endif
11609+#endif
11610+
11611+// Always turn off automatic FMA use - use STBIR_USE_FMA if you want.
11612+// Otherwise, this is a determinism disaster.
11613+#ifndef STBIR_DONT_CHANGE_FP_CONTRACT // override in case you don't want this behavior
11614+#if defined(_MSC_VER) && !defined(__clang__)
11615+#if _MSC_VER > 1200
11616+#pragma fp_contract(off)
11617+#endif
11618+#elif defined(__GNUC__) && !defined(__clang__)
11619+#pragma GCC optimize("fp-contract=off")
11620+#else
11621+#pragma STDC FP_CONTRACT OFF
11622+#endif
11623+#endif
11624+
11625+#ifdef _MSC_VER
11626+#define STBIR__UNUSED(v) (void)(v)
11627+#else
11628+#define STBIR__UNUSED(v) (void)sizeof(v)
11629+#endif
11630+
11631+#define STBIR__ARRAY_SIZE(a) (sizeof((a))/sizeof((a)[0]))
11632+
11633+
11634+#ifndef STBIR_DEFAULT_FILTER_UPSAMPLE
11635+#define STBIR_DEFAULT_FILTER_UPSAMPLE STBIR_FILTER_CATMULLROM
11636+#endif
11637+
11638+#ifndef STBIR_DEFAULT_FILTER_DOWNSAMPLE
11639+#define STBIR_DEFAULT_FILTER_DOWNSAMPLE STBIR_FILTER_MITCHELL
11640+#endif
11641+
11642+
11643+#ifndef STBIR__HEADER_FILENAME
11644+#define STBIR__HEADER_FILENAME "stb_image_resize2.h"
11645+#endif
11646+
11647+// the internal pixel layout enums are in a different order, so we can easily do range comparisons of types
11648+// the public pixel layout is ordered in a way that if you cast num_channels (1-4) to the enum, you get something sensible
11649+typedef enum
11650+{
11651+ STBIRI_1CHANNEL = 0,
11652+ STBIRI_2CHANNEL = 1,
11653+ STBIRI_RGB = 2,
11654+ STBIRI_BGR = 3,
11655+ STBIRI_4CHANNEL = 4,
11656+
11657+ STBIRI_RGBA = 5,
11658+ STBIRI_BGRA = 6,
11659+ STBIRI_ARGB = 7,
11660+ STBIRI_ABGR = 8,
11661+ STBIRI_RA = 9,
11662+ STBIRI_AR = 10,
11663+
11664+ STBIRI_RGBA_PM = 11,
11665+ STBIRI_BGRA_PM = 12,
11666+ STBIRI_ARGB_PM = 13,
11667+ STBIRI_ABGR_PM = 14,
11668+ STBIRI_RA_PM = 15,
11669+ STBIRI_AR_PM = 16,
11670+} stbir_internal_pixel_layout;
11671+
11672+// define the public pixel layouts to not compile inside the implementation (to avoid accidental use)
11673+#define STBIR_BGR bad_dont_use_in_implementation
11674+#define STBIR_1CHANNEL STBIR_BGR
11675+#define STBIR_2CHANNEL STBIR_BGR
11676+#define STBIR_RGB STBIR_BGR
11677+#define STBIR_RGBA STBIR_BGR
11678+#define STBIR_4CHANNEL STBIR_BGR
11679+#define STBIR_BGRA STBIR_BGR
11680+#define STBIR_ARGB STBIR_BGR
11681+#define STBIR_ABGR STBIR_BGR
11682+#define STBIR_RA STBIR_BGR
11683+#define STBIR_AR STBIR_BGR
11684+#define STBIR_RGBA_PM STBIR_BGR
11685+#define STBIR_BGRA_PM STBIR_BGR
11686+#define STBIR_ARGB_PM STBIR_BGR
11687+#define STBIR_ABGR_PM STBIR_BGR
11688+#define STBIR_RA_PM STBIR_BGR
11689+#define STBIR_AR_PM STBIR_BGR
11690+
11691+// must match stbir_datatype
11692+static unsigned char stbir__type_size[] = {
11693+ 1,1,1,2,4,2 // STBIR_TYPE_UINT8,STBIR_TYPE_UINT8_SRGB,STBIR_TYPE_UINT8_SRGB_ALPHA,STBIR_TYPE_UINT16,STBIR_TYPE_FLOAT,STBIR_TYPE_HALF_FLOAT
11694+};
11695+
11696+// When gathering, the contributors are which source pixels contribute.
11697+// When scattering, the contributors are which destination pixels are contributed to.
11698+typedef struct
11699+{
11700+ int n0; // First contributing pixel
11701+ int n1; // Last contributing pixel
11702+} stbir__contributors;
11703+
11704+typedef struct
11705+{
11706+ int lowest; // First sample index for whole filter
11707+ int highest; // Last sample index for whole filter
11708+ int widest; // widest single set of samples for an output
11709+} stbir__filter_extent_info;
11710+
11711+typedef struct stbir__scale_info
11712+{
11713+ int input_full_size;
11714+ int output_sub_size;
11715+ stbir_real scale;
11716+ stbir_real inv_scale;
11717+ stbir_real pixel_shift; // starting shift in output pixel space (in pixels)
11718+ int scale_is_rational;
11719+ stbir_uint32 scale_numerator, scale_denominator;
11720+ stbir_real tap_scale_in, tap_scale_out; // 1 / 2N and 1 / 2D for stbir__tap_offset
11721+ int positions_exact; // the tap positions come from the exact fraction and an integer shift (stbir__tap_offset)
11722+ int shift_pixels; // pixel_shift as that integer
11723+ int approximated_drift; // the fraction approximates the scale: how far its far edge can land from the float chain's, in input pixels (0: exact)
11724+} stbir__scale_info;
11725+
11726+typedef struct
11727+{
11728+ stbir__contributors * contributors;
11729+ float* coefficients;
11730+ stbir__scale_info scale_info;
11731+ stbir_filter filter_enum;
11732+ stbir__kernel_callback_new * filter_kernel_new; // the kernel: the built-in's or the user's, or
11733+ stbir__radius_callback_new * filter_radius; // the filter's extent, one of the three: a radius in input pixels (the box) or
11734+ stbir__support_callback_new * filter_support_new; // a support in filter pixels (the other built-ins); the user's, or the built-in's,
11735+ stbir__kernel_callback * filter_kernel_old; // an old back compat float kernel
11736+ stbir__support_callback * filter_support_old; // or an old back compat support function
11737+ stbir_edge edge;
11738+ int coefficient_width;
11739+ int entry_floats; // horizontal only: floats per packed column entry (STBIR__COLUMN_ENTRY_FLOATS of coefficient_width)
11740+
11741+ // polyphase sharing (see stbir__plan_polyphase_sharing). The coefficient sets for the repeating part of a rational scale
11742+ // are stored once. 0 = every output has its own set.
11743+ int shared_period; // coefficient sets in the repeating block
11744+ int shared_prologue; // outputs with their own coefficient sets before it
11745+ int shared_epilogue; // outputs with their own coefficient sets after it
11746+ int shared_block_n0; // n0 of the block's first output (output P), and how far n0 moves per pass over the block
11747+ int shared_block_shift;
11748+ int shared_epilogue_n0; // n0 of the first epilogue output
11749+ int filter_pixel_width;
11750+ int filter_pixel_margin;
11751+ int num_contributors;
11752+ size_t contributors_size;
11753+ size_t coefficients_size;
11754+ stbir__filter_extent_info extent_info;
11755+ int is_gather; // 0 = scatter, 1 = gather with scale >= 1, 2 = gather with scale < 1
11756+} stbir__sampler;
11757+
11758+// horizontal routines combine the contribs and coeffs into one structure - coefficients is actually an array of width floats.
11759+typedef struct
11760+{
11761+ int advance; // how far the running decode pointer moves for the NEXT column, in floats (already times the channel count)
11762+ int width; // how many pixels this column reads (n1 - n0 + 1) - only the n-coefficient kernels look at it
11763+ float coefficients[ 1 ]; // width floats long, with a zero pad after for the widths whose kernels load one float past
11764+} stbir__column_entry;
11765+
11766+// a column slice is a run of output columns that the whole vertical/horizontal loop is run over at once (instead of
11767+// full scanlines), so the ring buffer and the temp buffers stay in cache. the coefficients are shared by all the
11768+// slices, each slice just starts at a different offset into them.
11769+typedef struct
11770+{
11771+ int x0; // first output pixel of the slice
11772+ int width; // output pixels in the slice
11773+ int in_n0, in_n1; // input pixels decoded for the slice (in the horizontal sampler's coordinate space - can hang off either edge with WRAP)
11774+ int first_n0; // input pixel output x0 starts reading at
11775+} stbir__column_slice;
11776+
11777+typedef struct
11778+{
11779+#ifdef STBIR_PROFILE
11780+ union
11781+ {
11782+ struct { stbir_uint64 total, looping, vertical, horizontal, decode, encode, alpha, unalpha; } named;
11783+ stbir_uint64 array[8];
11784+ } profile;
11785+ stbir_uint64 * current_zone_excluded_ptr;
11786+#endif
11787+ float* decode_buffer;
11788+
11789+ int ring_buffer_first_scanline;
11790+ int ring_buffer_last_scanline;
11791+ int ring_buffer_begin_index; // first_scanline is at this index in the ring buffer
11792+ int start_output_y, end_output_y;
11793+ int start_input_y, end_input_y; // used in scatter only
11794+ int first_column_slice, num_column_slices; // the column slices this split runs: all of them when the splits are bands of rows, its own range when they are columns
11795+
11796+ #ifdef STBIR__SEPARATE_ALLOCATIONS
11797+ float** ring_buffers; // one pointer for each ring buffer
11798+ #else
11799+ float* ring_buffer; // one big buffer that we index into
11800+ #endif
11801+
11802+ float* vertical_buffer;
11803+
11804+ // where the buffers above were allocated. they each have a page of slack, and get repositioned within it at
11805+ // resize time to keep them clear of 4K aliasing with the input rows (see stbir__4K_alias_skew_buffers)
11806+ void* decode_buffer_alloc;
11807+ void* ring_buffer_alloc;
11808+ void* vertical_buffer_alloc;
11809+
11810+ char no_cache_straddle[64];
11811+} stbir__per_split_info;
11812+
11813+typedef float * stbir__decode_pixels_func( float * decode, ptrdiff_t width_times_channels, void const * input );
11814+typedef void stbir__alpha_weight_func( float * decode_buffer, ptrdiff_t width_times_channels );
11815+typedef void stbir__horizontal_gather_channels_func( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer,
11816+ stbir__column_entry const * column, int entry_stride );
11817+typedef void stbir__alpha_unweight_func(float * encode_buffer, ptrdiff_t width_times_channels );
11818+typedef void stbir__encode_pixels_func( void * output, ptrdiff_t width_times_channels, float const * encode );
11819+
11820+struct stbir__info
11821+{
11822+#ifdef STBIR_PROFILE
11823+ union
11824+ {
11825+ struct { stbir_uint64 total, build, alloc, horizontal, vertical, cleanup, scatter; } named;
11826+ stbir_uint64 array[7];
11827+ } profile;
11828+ stbir_uint64 * current_zone_excluded_ptr;
11829+#endif
11830+ stbir__sampler horizontal;
11831+ stbir__sampler vertical;
11832+
11833+ void const * input_data;
11834+ void * output_data;
11835+
11836+ stbir_int64 input_stride_bytes;
11837+ stbir_int64 output_stride_bytes;
11838+ size_t ring_buffer_length_bytes; // The length of an individual entry in the ring buffer (there are ring_buffer_num_entries of them)
11839+ int ring_buffer_num_entries; // Total number of entries in the ring buffer.
11840+
11841+ stbir_datatype input_type;
11842+ stbir_datatype output_type;
11843+
11844+ stbir_input_callback * in_pixels_cb;
11845+ void * user_data;
11846+ stbir_output_callback * out_pixels_cb;
11847+
11848+ stbir__contributors conservative_extents; // input pixel range that covers everything the horizontal filter can touch (decode buffers are sized from this)
11849+
11850+ void * alloced_mem;
11851+ stbir__per_split_info * split_info; // by default 1, but there will be N of these allocated based on the thread init you did
11852+ stbir__column_slice * column_slices; // the output columns, left to right (the last slice can be narrower)
11853+ int num_column_slices;
11854+ int split_cols, split_rows; // the split grid: splits = split_cols column groups x split_rows row bands (see stbir__split_mode)
11855+
11856+ stbir__decode_pixels_func * decode_pixels;
11857+ stbir__alpha_weight_func * alpha_weight;
11858+ stbir__horizontal_gather_channels_func * horizontal_gather_channels;
11859+ stbir__horizontal_gather_channels_func * horizontal_gather_period1; // for the run over a polyphase period 1 set (integer downsamples: one coefficient set for every output), or null
11860+ stbir__alpha_unweight_func * alpha_unweight;
11861+ stbir__encode_pixels_func * encode_pixels;
11862+
11863+ int alloc_ring_buffer_num_entries; // Number of entries in the ring buffer that will be allocated
11864+ int splits; // count of splits
11865+
11866+ stbir_internal_pixel_layout input_pixel_layout_internal;
11867+ stbir_internal_pixel_layout output_pixel_layout_internal;
11868+
11869+ int input_color_and_type;
11870+ int input_callback_offset_bytes; // where the input callback's temp buffer sits relative to the decode target (see STBIR_INPUT_CALLBACK_PADDING)
11871+ int offset_x, offset_y; // offset within output_data
11872+ int vertical_first;
11873+ int channels;
11874+ int effective_channels; // same as channels, except on RGBA/ARGB (7), or XA/AX (3)
11875+ size_t alloced_total;
11876+};
11877+
11878+
11879+#define stbir__max_uint8_as_float 255.0f
11880+#define stbir__max_uint16_as_float 65535.0f
11881+#define stbir__max_uint8_as_float_inverted 3.9215689e-03f // (1.0f/255.0f)
11882+#define stbir__max_uint16_as_float_inverted 1.5259022e-05f // (1.0f/65535.0f)
11883+
11884+// an alpha below this isn't divided by when un-premultiplying - alpha is effectively zero
11885+#define stbir__alpha_too_small_to_divide ((float)1 / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20))
11886+
11887+// a coefficient set whose sum is within this of 1 isn't renormalized: scaling by 1 / sum would move every float
11888+// coefficient by less than half an ulp ( 2^-25 of itself ), so each would round straight back - the loop is a no-op
11889+#define stbir__renorm_tolerance ( (stbir_real)1.0 / ( 1 << 26 ) )
11890+
11891+// a kernel value this small (either sign) is dropped before the coefficients are normalized
11892+#define stbir__too_small_coefficient ( (stbir_real)1.0 / ( 1 << 26 ) )
11893+
11894+// a coefficient set whose sum is under this (either sign) is zeroed instead of normalized
11895+#define stbir__too_small_coefficient_sum ( (stbir_real)1.0 / ( 1 << 16 ) )
11896+
11897+// min/max friendly
11898+#define STBIR_CLAMP(x, xmin, xmax) for(;;) { \
11899+ if ( (x) < (xmin) ) (x) = (xmin); \
11900+ if ( (x) > (xmax) ) (x) = (xmax); \
11901+ break; \
11902+}
11903+
11904+static stbir__inline int stbir__min(int a, int b)
11905+{
11906+ return a < b ? a : b;
11907+}
11908+
11909+static stbir__inline int stbir__max(int a, int b)
11910+{
11911+ return a > b ? a : b;
11912+}
11913+
11914+static float stbir__srgb_uchar_to_linear_float[256] = {
11915+ 0.000000f, 0.000304f, 0.000607f, 0.000911f, 0.001214f, 0.001518f, 0.001821f, 0.002125f, 0.002428f, 0.002732f, 0.003035f,
11916+ 0.003347f, 0.003677f, 0.004025f, 0.004391f, 0.004777f, 0.005182f, 0.005605f, 0.006049f, 0.006512f, 0.006995f, 0.007499f,
11917+ 0.008023f, 0.008568f, 0.009134f, 0.009721f, 0.010330f, 0.010960f, 0.011612f, 0.012286f, 0.012983f, 0.013702f, 0.014444f,
11918+ 0.015209f, 0.015996f, 0.016807f, 0.017642f, 0.018500f, 0.019382f, 0.020289f, 0.021219f, 0.022174f, 0.023153f, 0.024158f,
11919+ 0.025187f, 0.026241f, 0.027321f, 0.028426f, 0.029557f, 0.030713f, 0.031896f, 0.033105f, 0.034340f, 0.035601f, 0.036889f,
11920+ 0.038204f, 0.039546f, 0.040915f, 0.042311f, 0.043735f, 0.045186f, 0.046665f, 0.048172f, 0.049707f, 0.051269f, 0.052861f,
11921+ 0.054480f, 0.056128f, 0.057805f, 0.059511f, 0.061246f, 0.063010f, 0.064803f, 0.066626f, 0.068478f, 0.070360f, 0.072272f,
11922+ 0.074214f, 0.076185f, 0.078187f, 0.080220f, 0.082283f, 0.084376f, 0.086500f, 0.088656f, 0.090842f, 0.093059f, 0.095307f,
11923+ 0.097587f, 0.099899f, 0.102242f, 0.104616f, 0.107023f, 0.109462f, 0.111932f, 0.114435f, 0.116971f, 0.119538f, 0.122139f,
11924+ 0.124772f, 0.127438f, 0.130136f, 0.132868f, 0.135633f, 0.138432f, 0.141263f, 0.144128f, 0.147027f, 0.149960f, 0.152926f,
11925+ 0.155926f, 0.158961f, 0.162029f, 0.165132f, 0.168269f, 0.171441f, 0.174647f, 0.177888f, 0.181164f, 0.184475f, 0.187821f,
11926+ 0.191202f, 0.194618f, 0.198069f, 0.201556f, 0.205079f, 0.208637f, 0.212231f, 0.215861f, 0.219526f, 0.223228f, 0.226966f,
11927+ 0.230740f, 0.234551f, 0.238398f, 0.242281f, 0.246201f, 0.250158f, 0.254152f, 0.258183f, 0.262251f, 0.266356f, 0.270498f,
11928+ 0.274677f, 0.278894f, 0.283149f, 0.287441f, 0.291771f, 0.296138f, 0.300544f, 0.304987f, 0.309469f, 0.313989f, 0.318547f,
11929+ 0.323143f, 0.327778f, 0.332452f, 0.337164f, 0.341914f, 0.346704f, 0.351533f, 0.356400f, 0.361307f, 0.366253f, 0.371238f,
11930+ 0.376262f, 0.381326f, 0.386430f, 0.391573f, 0.396755f, 0.401978f, 0.407240f, 0.412543f, 0.417885f, 0.423268f, 0.428691f,
11931+ 0.434154f, 0.439657f, 0.445201f, 0.450786f, 0.456411f, 0.462077f, 0.467784f, 0.473532f, 0.479320f, 0.485150f, 0.491021f,
11932+ 0.496933f, 0.502887f, 0.508881f, 0.514918f, 0.520996f, 0.527115f, 0.533276f, 0.539480f, 0.545725f, 0.552011f, 0.558340f,
11933+ 0.564712f, 0.571125f, 0.577581f, 0.584078f, 0.590619f, 0.597202f, 0.603827f, 0.610496f, 0.617207f, 0.623960f, 0.630757f,
11934+ 0.637597f, 0.644480f, 0.651406f, 0.658375f, 0.665387f, 0.672443f, 0.679543f, 0.686685f, 0.693872f, 0.701102f, 0.708376f,
11935+ 0.715694f, 0.723055f, 0.730461f, 0.737911f, 0.745404f, 0.752942f, 0.760525f, 0.768151f, 0.775822f, 0.783538f, 0.791298f,
11936+ 0.799103f, 0.806952f, 0.814847f, 0.822786f, 0.830770f, 0.838799f, 0.846873f, 0.854993f, 0.863157f, 0.871367f, 0.879622f,
11937+ 0.887923f, 0.896269f, 0.904661f, 0.913099f, 0.921582f, 0.930111f, 0.938686f, 0.947307f, 0.955974f, 0.964686f, 0.973445f,
11938+ 0.982251f, 0.991102f, 1.0f
11939+};
11940+
11941+typedef union
11942+{
11943+ unsigned int u;
11944+ float f;
11945+} stbir__FP32;
11946+
11947+// From https://gist.github.com/rygorous/2203834
11948+
11949+// the SIMD sRGB encoders index this with a float's bits munged into 912..1015, so this biases it back to the table size
11950+#define STBIR__SRGB8_TAB4_BIAS ((127-13)*8)
11951+#define STBIR__SRGB8_TAB4( index ) fp32_to_srgb8_tab4[ (ptrdiff_t)(index) - STBIR__SRGB8_TAB4_BIAS ]
11952+static const stbir_uint32 fp32_to_srgb8_tab4[104] = {
11953+ 0x0073000d, 0x007a000d, 0x0080000d, 0x0087000d, 0x008d000d, 0x0094000d, 0x009a000d, 0x00a1000d,
11954+ 0x00a7001a, 0x00b4001a, 0x00c1001a, 0x00ce001a, 0x00da001a, 0x00e7001a, 0x00f4001a, 0x0101001a,
11955+ 0x010e0033, 0x01280033, 0x01410033, 0x015b0033, 0x01750033, 0x018f0033, 0x01a80033, 0x01c20033,
11956+ 0x01dc0067, 0x020f0067, 0x02430067, 0x02760067, 0x02aa0067, 0x02dd0067, 0x03110067, 0x03440067,
11957+ 0x037800ce, 0x03df00ce, 0x044600ce, 0x04ad00ce, 0x051400ce, 0x057b00c5, 0x05dd00bc, 0x063b00b5,
11958+ 0x06970158, 0x07420142, 0x07e30130, 0x087b0120, 0x090b0112, 0x09940106, 0x0a1700fc, 0x0a9500f2,
11959+ 0x0b0f01cb, 0x0bf401ae, 0x0ccb0195, 0x0d950180, 0x0e56016e, 0x0f0d015e, 0x0fbc0150, 0x10630143,
11960+ 0x11070264, 0x1238023e, 0x1357021d, 0x14660201, 0x156601e9, 0x165a01d3, 0x174401c0, 0x182401af,
11961+ 0x18fe0331, 0x1a9602fe, 0x1c1502d2, 0x1d7e02ad, 0x1ed4028d, 0x201a0270, 0x21520256, 0x227d0240,
11962+ 0x239f0443, 0x25c003fe, 0x27bf03c4, 0x29a10392, 0x2b6a0367, 0x2d1d0341, 0x2ebe031f, 0x304d0300,
11963+ 0x31d105b0, 0x34a80555, 0x37520507, 0x39d504c5, 0x3c37048b, 0x3e7c0458, 0x40a8042a, 0x42bd0401,
11964+ 0x44c20798, 0x488e071e, 0x4c1c06b6, 0x4f76065d, 0x52a50610, 0x55ac05cc, 0x5892058f, 0x5b590559,
11965+ 0x5e0c0a23, 0x631c0980, 0x67db08f6, 0x6c55087f, 0x70940818, 0x74a007bd, 0x787d076c, 0x7c330723,
11966+};
11967+
11968+static stbir__inline stbir_uint8 stbir__linear_to_srgb_uchar(float in)
11969+{
11970+ static const stbir__FP32 almostone = { 0x3f7fffff }; // 1-eps
11971+ static const stbir__FP32 minval = { (127-13) << 23 };
11972+ stbir_uint32 tab,bias,scale,t;
11973+ stbir__FP32 f;
11974+
11975+ // Clamp to [2^(-13), 1-eps]; these two values map to 0 and 1, respectively.
11976+ // The tests are carefully written so that NaNs map to 0, same as in the reference
11977+ // implementation.
11978+ if (!(in > minval.f)) // written this way to catch NaNs
11979+ return 0;
11980+ if (in > almostone.f)
11981+ return 255;
11982+
11983+ // Do the table lookup and unpack bias, scale
11984+ f.f = in;
11985+ tab = fp32_to_srgb8_tab4[(f.u - minval.u) >> 20];
11986+ bias = (tab >> 16) << 9;
11987+ scale = tab & 0xffff;
11988+
11989+ // Grab next-highest mantissa bits and perform linear interpolation
11990+ t = (f.u >> 12) & 0xff;
11991+ return (unsigned char) ((bias + scale*t) >> 16);
11992+}
11993+
11994+#ifndef STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT
11995+#define STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT 128 // when downsampling and <= 128 scanlines of buffering, use gather (scatter above that - 32 or more to 1 downsampling)
11996+#endif
11997+
11998+
11999+// every scanline buffer carries this many extra floats past the end. we right justify into the buffer
12000+// so the conversion can read ahead of the writes. also gives the horizontal kernels a one-float overread
12001+#define STBIR_INPUT_CALLBACK_PADDING 16
12002+
12003+#ifdef _M_IX86_FP
12004+#if ( _M_IX86_FP >= 2 )
12005+#ifndef STBIR_SSE
12006+#define STBIR_SSE
12007+#endif
12008+#endif
12009+#endif
12010+
12011+#ifdef __TINYC__
12012+ // tiny c has no intrinsics yet - this can become a version check if they add them
12013+ #define STBIR_NO_SIMD
12014+#endif
12015+
12016+// (ARM64EC defines _M_X64 and _M_AMD64 as well as _M_ARM64EC, so the x64 tests have to leave it out - it is NEON below)
12017+#if ( defined(_x86_64) || defined( __x86_64__ ) || defined( _M_X64 ) || defined(__x86_64) || defined(_M_AMD64) || defined(__SSE2__) || defined(STBIR_SSE) || defined(STBIR_SSE2) ) && !defined(_M_ARM64EC)
12018+ #ifndef STBIR_SSE2
12019+ #define STBIR_SSE2
12020+ #endif
12021+ // AVX-512: the foundation plus the byte/word (BW) and 128/256-bit (VL) extensions. /arch:AVX512 on microsoft,
12022+ // -mavx512f -mavx512bw -mavx512vl (or -march=x86-64-v4) on clang and gcc. Also implies AVX2
12023+ #if ( defined(__AVX512F__) && defined(__AVX512BW__) && defined(__AVX512VL__) ) || defined(STBIR_AVX512)
12024+ #ifdef STBIR_NO_AVX512
12025+ #undef STBIR_AVX512
12026+ #else
12027+ #ifndef STBIR_AVX512
12028+ #define STBIR_AVX512
12029+ #endif
12030+ #ifndef STBIR_AVX2
12031+ #define STBIR_AVX2
12032+ #endif
12033+ #endif
12034+ #endif
12035+ #if defined(__AVX__) || defined(STBIR_AVX2)
12036+ #ifndef STBIR_AVX
12037+ #ifndef STBIR_NO_AVX
12038+ #define STBIR_AVX
12039+ #endif
12040+ #endif
12041+ #endif
12042+ #if defined(__AVX2__) || defined(STBIR_AVX2)
12043+ #ifndef STBIR_NO_AVX2
12044+ #ifndef STBIR_AVX2
12045+ #define STBIR_AVX2
12046+ #endif
12047+ #if defined( _MSC_VER ) && !defined(__clang__)
12048+ #ifndef STBIR_FP16C // FP16C instructions are on all AVX2 cpus, so we can autoselect it here on microsoft - clang needs -mf16c
12049+ #define STBIR_FP16C
12050+ #endif
12051+ #endif
12052+ #endif
12053+ #endif
12054+ #ifdef __F16C__
12055+ #ifndef STBIR_FP16C // turn on FP16C instructions if the define is set (for clang and gcc)
12056+ #define STBIR_FP16C
12057+ #endif
12058+ #endif
12059+#endif
12060+
12061+// 64-bit ARM (MSVC's ARM64EC included): the NEON code has a few AArch64 only forms
12062+#if defined( _M_ARM64 ) || defined( _M_ARM64EC ) || defined( __aarch64__ ) || defined( __arm64__ )
12063+#define STBIR__ARM64
12064+#endif
12065+
12066+#if defined( STBIR__ARM64 ) || ((__ARM_NEON_FP & 4) != 0) || defined(__ARM_NEON__)
12067+#ifndef STBIR_NEON
12068+#define STBIR_NEON
12069+#endif
12070+#endif
12071+
12072+#if defined(_M_ARM) || defined(__arm__)
12073+#ifdef STBIR_USE_FMA
12074+#undef STBIR_USE_FMA // no FMA for 32-bit arm on MSVC
12075+#endif
12076+#endif
12077+
12078+#if defined(__wasm__) && defined(__wasm_simd128__)
12079+#ifndef STBIR_WASM
12080+#define STBIR_WASM
12081+#endif
12082+#endif
12083+
12084+// restrict pointers for the output pointers, other loop and unroll control
12085+#if defined( _MSC_VER ) && !defined(__clang__)
12086+ #define STBIR_STREAMOUT_PTR( star ) star __restrict
12087+ #define STBIR_NO_UNROLL( ptr ) __assume(ptr) // this oddly keeps msvc from unrolling a loop
12088+ #if _MSC_VER >= 1900
12089+ #define STBIR_NO_UNROLL_LOOP_START __pragma(loop( no_vector ))
12090+ #else
12091+ #define STBIR_NO_UNROLL_LOOP_START
12092+ #endif
12093+#elif defined( __clang__ )
12094+ #define STBIR_STREAMOUT_PTR( star ) star __restrict__
12095+ #define STBIR_NO_UNROLL( ptr ) __asm__ (""::"r"(ptr))
12096+ #if ( __clang_major__ >= 4 ) || ( ( __clang_major__ >= 3 ) && ( __clang_minor__ >= 5 ) )
12097+ #define STBIR_NO_UNROLL_LOOP_START _Pragma("clang loop unroll(disable)") _Pragma("clang loop vectorize(disable)")
12098+ #else
12099+ #define STBIR_NO_UNROLL_LOOP_START
12100+ #endif
12101+#elif defined( __GNUC__ )
12102+ #define STBIR_STREAMOUT_PTR( star ) star __restrict__
12103+ #define STBIR_NO_UNROLL( ptr ) __asm__ (""::"r"(ptr))
12104+ #if __GNUC__ >= 14
12105+ #define STBIR_NO_UNROLL_LOOP_START _Pragma("GCC unroll 0") _Pragma("GCC novector")
12106+ #else
12107+ #define STBIR_NO_UNROLL_LOOP_START
12108+ #endif
12109+ #define STBIR_NO_UNROLL_LOOP_START_INF_FOR
12110+#else
12111+ #define STBIR_STREAMOUT_PTR( star ) star
12112+ #define STBIR_NO_UNROLL( ptr )
12113+ #define STBIR_NO_UNROLL_LOOP_START
12114+#endif
12115+
12116+#ifndef STBIR_NO_UNROLL_LOOP_START_INF_FOR
12117+#define STBIR_NO_UNROLL_LOOP_START_INF_FOR STBIR_NO_UNROLL_LOOP_START
12118+#endif
12119+
12120+#ifdef STBIR_NO_SIMD // force simd off for whatever reason
12121+
12122+// force simd off overrides everything else, so clear it all
12123+
12124+#ifdef STBIR_SSE2
12125+#undef STBIR_SSE2
12126+#endif
12127+
12128+#ifdef STBIR_AVX
12129+#undef STBIR_AVX
12130+#endif
12131+
12132+#ifdef STBIR_NEON
12133+#undef STBIR_NEON
12134+#endif
12135+
12136+#ifdef STBIR_AVX2
12137+#undef STBIR_AVX2
12138+#endif
12139+
12140+#ifdef STBIR_FP16C
12141+#undef STBIR_FP16C
12142+#endif
12143+
12144+#ifdef STBIR_WASM
12145+#undef STBIR_WASM
12146+#endif
12147+
12148+#ifdef STBIR_SIMD
12149+#undef STBIR_SIMD
12150+#endif
12151+
12152+#else // STBIR_SIMD
12153+
12154+#ifdef STBIR_SSE2
12155+ #include <emmintrin.h>
12156+
12157+ #define stbir__simdf __m128
12158+ #define stbir__simdi __m128i
12159+
12160+ #define stbir_simdi_castf( reg ) _mm_castps_si128(reg)
12161+ #define stbir_simdf_casti( reg ) _mm_castsi128_ps(reg)
12162+
12163+ #define stbir__simdf_load( reg, ptr ) (reg) = _mm_loadu_ps( (float const*)(ptr) )
12164+ #define stbir__simdi_load( reg, ptr ) (reg) = _mm_loadu_si128 ( (stbir__simdi const*)(ptr) )
12165+ #define stbir__simdf_load1( out, ptr ) (out) = _mm_load_ss( (float const*)(ptr) ) // top values can be random (not denormal or nan for perf)
12166+ #define stbir__simdf_load1z( out, ptr ) (out) = _mm_load_ss( (float const*)(ptr) ) // top values must be zero
12167+ #define stbir__simdf_frep4( fvar ) _mm_set_ps1( fvar )
12168+ #define stbir__simdf_load1frep4( out, fvar ) (out) = _mm_set_ps1( fvar )
12169+ #define stbir__simdf_load2( out, ptr ) (out) = _mm_castsi128_ps( _mm_loadl_epi64( (__m128i*)(ptr)) ) // top values can be random (not denormal or nan for perf)
12170+ #define stbir__simdf_load2z( out, ptr ) (out) = _mm_castsi128_ps( _mm_loadl_epi64( (__m128i*)(ptr)) ) // top values must be zero
12171+ #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = _mm_castpd_ps(_mm_loadh_pd( _mm_castps_pd(reg), (double*)(ptr) ))
12172+ // pairwise add of two registers: out = ( a0+a1, a2+a3, b0+b1, b2+b3 ), and the halves version: ( a0+a2, a1+a3, b0+b2, b1+b3 )
12173+ #define stbir__simdf_hadd2( out, a, b ) (out) = _mm_add_ps( _mm_shuffle_ps( a, b, (0<<0)+(2<<2)+(0<<4)+(2<<6) ), _mm_shuffle_ps( a, b, (1<<0)+(3<<2)+(1<<4)+(3<<6) ) )
12174+ #define stbir__simdf_addhalves2( out, a, b ) (out) = _mm_add_ps( _mm_movelh_ps( a, b ), _mm_movehl_ps( b, a ) )
12175+
12176+ #define stbir__simdf_zeroP() _mm_setzero_ps()
12177+ #define stbir__simdf_zero( reg ) (reg) = _mm_setzero_ps()
12178+
12179+ #define stbir__simdf_store( ptr, reg ) _mm_storeu_ps( (float*)(ptr), reg )
12180+ #define stbir__simdf_store1( ptr, reg ) _mm_store_ss( (float*)(ptr), reg )
12181+ #define stbir__simdf_store2( ptr, reg ) _mm_storel_epi64( (__m128i*)(ptr), _mm_castps_si128(reg) )
12182+ #define stbir__simdf_store2h( ptr, reg ) _mm_storeh_pd( (double*)(ptr), _mm_castps_pd(reg) )
12183+
12184+ #define stbir__simdi_store( ptr, reg ) _mm_storeu_si128( (__m128i*)(ptr), reg )
12185+ #define stbir__simdi_store2( ptr, reg ) _mm_storel_epi64( (__m128i*)(ptr), (reg) )
12186+
12187+ #define stbir__prefetch( ptr ) _mm_prefetch((char*)(ptr), _MM_HINT_T0 )
12188+
12189+ #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
12190+ { \
12191+ stbir__simdi zero = _mm_setzero_si128(); \
12192+ out2 = _mm_unpacklo_epi8( ireg, zero ); \
12193+ out3 = _mm_unpackhi_epi8( ireg, zero ); \
12194+ out0 = _mm_unpacklo_epi16( out2, zero ); \
12195+ out1 = _mm_unpackhi_epi16( out2, zero ); \
12196+ out2 = _mm_unpacklo_epi16( out3, zero ); \
12197+ out3 = _mm_unpackhi_epi16( out3, zero ); \
12198+ }
12199+
12200+
12201+ #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
12202+ { \
12203+ stbir__simdi zero = _mm_setzero_si128(); \
12204+ out0 = _mm_unpacklo_epi16( ireg, zero ); \
12205+ out1 = _mm_unpackhi_epi16( ireg, zero ); \
12206+ }
12207+
12208+ #define stbir__simdf_convert_float_to_i32( i, f ) (i) = _mm_cvttps_epi32(f)
12209+ #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)_mm_cvtsi128_si32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(f,STBIR__CONSTF(STBIR_max_uint8_as_float)),_mm_setzero_ps()))))
12210+ #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)_mm_cvtsi128_si32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(f,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps()))))
12211+
12212+ #define stbir__simdi_to_int( i ) _mm_cvtsi128_si32(i)
12213+ #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = _mm_cvtepi32_ps( ireg )
12214+ #define stbir__simdf_add( out, reg0, reg1 ) (out) = _mm_add_ps( reg0, reg1 )
12215+ #define stbir__simdf_mult( out, reg0, reg1 ) (out) = _mm_mul_ps( reg0, reg1 )
12216+ #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = _mm_mul_ps( reg, _mm_loadu_ps( (float const*)(ptr) ) )
12217+ #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = _mm_mul_ss( reg, _mm_load_ss( (float const*)(ptr) ) )
12218+ #define stbir__simdf_add_mem( out, reg, ptr ) (out) = _mm_add_ps( reg, _mm_loadu_ps( (float const*)(ptr) ) )
12219+
12220+ #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd
12221+ #include <immintrin.h>
12222+ #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = _mm_fmadd_ps( mul1, mul2, add )
12223+ #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = _mm_fmadd_ps( mul, _mm_loadu_ps( (float const*)(ptr) ), add )
12224+ #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = _mm_fmadd_ss( mul, _mm_load_ss( (float const*)(ptr) ), add )
12225+ #else
12226+ #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = _mm_add_ps( add, _mm_mul_ps( mul1, mul2 ) )
12227+ #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = _mm_add_ps( add, _mm_mul_ps( mul, _mm_loadu_ps( (float const*)(ptr) ) ) )
12228+ #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = _mm_add_ss( add, _mm_mul_ss( mul, _mm_load_ss( (float const*)(ptr) ) ) )
12229+ #endif
12230+
12231+ #define stbir__simdf_add1( out, reg0, reg1 ) (out) = _mm_add_ss( reg0, reg1 )
12232+
12233+ #define stbir__simdf_and( out, reg0, reg1 ) (out) = _mm_and_ps( reg0, reg1 )
12234+
12235+ #define stbir__simdf_min( out, reg0, reg1 ) (out) = _mm_min_ps( reg0, reg1 )
12236+ #define stbir__simdf_max( out, reg0, reg1 ) (out) = _mm_max_ps( reg0, reg1 )
12237+
12238+ #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_shuffle_ps( reg1,reg0, (0<<0) + (1<<2) + (2<<4) + (3<<6) )), (3<<0) + (0<<2) + (1<<4) + (2<<6) ) )
12239+ #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_shuffle_ps( reg1,reg0, (0<<0) + (1<<2) + (2<<4) + (3<<6) )), (2<<0) + (3<<2) + (0<<4) + (1<<6) ) )
12240+
12241+ static const stbir__simdf STBIR_zeroones = { 0.0f,1.0f,0.0f,1.0f };
12242+ static const stbir__simdf STBIR_onezeros = { 1.0f,0.0f,1.0f,0.0f };
12243+ #define stbir__simdf_aaa1( out, alp, ones ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_movehl_ps( ones, alp ) ), (1<<0) + (1<<2) + (1<<4) + (2<<6) ) )
12244+ #define stbir__simdf_1aaa( out, alp, ones ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_movelh_ps( ones, alp ) ), (0<<0) + (2<<2) + (2<<4) + (2<<6) ) )
12245+ #define stbir__simdf_a1a1( out, alp, ones) (out) = _mm_or_ps( _mm_castsi128_ps( _mm_srli_epi64( _mm_castps_si128(alp), 32 ) ), STBIR_zeroones )
12246+ #define stbir__simdf_1a1a( out, alp, ones) (out) = _mm_or_ps( _mm_castsi128_ps( _mm_slli_epi64( _mm_castps_si128(alp), 32 ) ), STBIR_onezeros )
12247+
12248+ #define stbir__simdf_swiz( reg, one, two, three, four ) _mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( reg ), (one<<0) + (two<<2) + (three<<4) + (four<<6) ) )
12249+
12250+ #define stbir__simdi_and( out, reg0, reg1 ) (out) = _mm_and_si128( reg0, reg1 )
12251+ #define stbir__simdi_or( out, reg0, reg1 ) (out) = _mm_or_si128( reg0, reg1 )
12252+ #define stbir__simdi_16madd( out, reg0, reg1 ) (out) = _mm_madd_epi16( reg0, reg1 )
12253+
12254+ #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
12255+ { \
12256+ stbir__simdf af,bf; \
12257+ stbir__simdi a,b; \
12258+ af = _mm_min_ps( aa, STBIR_max_uint8_as_float ); \
12259+ bf = _mm_min_ps( bb, STBIR_max_uint8_as_float ); \
12260+ af = _mm_max_ps( af, _mm_setzero_ps() ); \
12261+ bf = _mm_max_ps( bf, _mm_setzero_ps() ); \
12262+ a = _mm_cvttps_epi32( af ); \
12263+ b = _mm_cvttps_epi32( bf ); \
12264+ a = _mm_packs_epi32( a, b ); \
12265+ out = _mm_packus_epi16( a, a ); \
12266+ }
12267+
12268+ #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
12269+ stbir__simdf_load( o0, (ptr) ); \
12270+ stbir__simdf_load( o1, (ptr)+4 ); \
12271+ stbir__simdf_load( o2, (ptr)+8 ); \
12272+ stbir__simdf_load( o3, (ptr)+12 ); \
12273+ { \
12274+ __m128 tmp0, tmp1, tmp2, tmp3; \
12275+ tmp0 = _mm_unpacklo_ps(o0, o1); \
12276+ tmp2 = _mm_unpacklo_ps(o2, o3); \
12277+ tmp1 = _mm_unpackhi_ps(o0, o1); \
12278+ tmp3 = _mm_unpackhi_ps(o2, o3); \
12279+ o0 = _mm_movelh_ps(tmp0, tmp2); \
12280+ o1 = _mm_movehl_ps(tmp2, tmp0); \
12281+ o2 = _mm_movelh_ps(tmp1, tmp3); \
12282+ o3 = _mm_movehl_ps(tmp3, tmp1); \
12283+ }
12284+
12285+ #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
12286+ r0 = _mm_packs_epi32( r0, r1 ); \
12287+ r2 = _mm_packs_epi32( r2, r3 ); \
12288+ r1 = _mm_unpacklo_epi16( r0, r2 ); \
12289+ r3 = _mm_unpackhi_epi16( r0, r2 ); \
12290+ r0 = _mm_unpacklo_epi16( r1, r3 ); \
12291+ r2 = _mm_unpackhi_epi16( r1, r3 ); \
12292+ r0 = _mm_packus_epi16( r0, r2 ); \
12293+ stbir__simdi_store( ptr, r0 ); \
12294+
12295+ #define stbir__simdi_32shr( out, reg, imm ) out = _mm_srli_epi32( reg, imm )
12296+
12297+ #if defined(_MSC_VER) && !defined(__clang__)
12298+ // msvc inits with 8 bytes
12299+ #define STBIR__CONST_32_TO_8( v ) (char)(unsigned char)((v)&255),(char)(unsigned char)(((v)>>8)&255),(char)(unsigned char)(((v)>>16)&255),(char)(unsigned char)(((v)>>24)&255)
12300+ #define STBIR__CONST_4_32i( v ) STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v )
12301+ #define STBIR__CONST_4d_32i( v0, v1, v2, v3 ) STBIR__CONST_32_TO_8( v0 ), STBIR__CONST_32_TO_8( v1 ), STBIR__CONST_32_TO_8( v2 ), STBIR__CONST_32_TO_8( v3 )
12302+ #else
12303+ // everything else inits with long long's
12304+ #define STBIR__CONST_4_32i( v ) (long long)((((stbir_uint64)(stbir_uint32)(v))<<32)|((stbir_uint64)(stbir_uint32)(v))),(long long)((((stbir_uint64)(stbir_uint32)(v))<<32)|((stbir_uint64)(stbir_uint32)(v)))
12305+ #define STBIR__CONST_4d_32i( v0, v1, v2, v3 ) (long long)((((stbir_uint64)(stbir_uint32)(v1))<<32)|((stbir_uint64)(stbir_uint32)(v0))),(long long)((((stbir_uint64)(stbir_uint32)(v3))<<32)|((stbir_uint64)(stbir_uint32)(v2)))
12306+ #endif
12307+
12308+ #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = { x, x, x, x }
12309+ #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { STBIR__CONST_4_32i(x) }
12310+ #define STBIR__SIMDI_CONST4(var, a, b, c, d) stbir__simdi var = { STBIR__CONST_4d_32i(a, b, c, d) }
12311+ #define STBIR__CONSTF(var) (var)
12312+ #define STBIR__CONSTI(var) (var)
12313+
12314+ // an arbitrary byte permute (pshufb) needs SSSE3
12315+ #if defined(STBIR_AVX) || defined(__SSSE3__) || defined(__SSE4_1__)
12316+ #include <tmmintrin.h>
12317+ #define STBIR_SIMD_BYTE_SHUFFLE
12318+ #define stbir__simdi_shuffle_bytes( out, reg, table ) (out) = _mm_shuffle_epi8( (reg), (table) )
12319+ #endif
12320+
12321+ #if defined(STBIR_AVX) || defined(__SSE4_1__)
12322+ #include <smmintrin.h>
12323+ #define stbir__simdf_pack_to_8words(out,reg0,reg1) out = _mm_packus_epi32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg0,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())), _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg1,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())))
12324+ #else
12325+ static STBIR__SIMDI_CONST(stbir__s32_32768, 32768);
12326+ static STBIR__SIMDI_CONST(stbir__s16_32768, ((32768<<16)|32768));
12327+
12328+ #define stbir__simdf_pack_to_8words(out,reg0,reg1) \
12329+ { \
12330+ stbir__simdi tmp0,tmp1; \
12331+ tmp0 = _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg0,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())); \
12332+ tmp1 = _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg1,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())); \
12333+ tmp0 = _mm_sub_epi32( tmp0, stbir__s32_32768 ); \
12334+ tmp1 = _mm_sub_epi32( tmp1, stbir__s32_32768 ); \
12335+ out = _mm_packs_epi32( tmp0, tmp1 ); \
12336+ out = _mm_sub_epi16( out, stbir__s16_32768 ); \
12337+ }
12338+
12339+ #endif
12340+
12341+ #define STBIR_SIMD
12342+
12343+ // if we detect AVX, set the simd8 defines
12344+ #ifdef STBIR_AVX
12345+ #include <immintrin.h>
12346+ #define STBIR_SIMD8
12347+ #define stbir__simdf8 __m256
12348+ #define stbir__simdi8 __m256i
12349+ #define stbir__simdf8_load( out, ptr ) (out) = _mm256_loadu_ps( (float const *)(ptr) )
12350+ #define stbir__simdi8_load( out, ptr ) (out) = _mm256_loadu_si256( (__m256i const *)(ptr) )
12351+ #define stbir__simdf8_mult( out, a, b ) (out) = _mm256_mul_ps( (a), (b) )
12352+ #define stbir__simdf8_store( ptr, out ) _mm256_storeu_ps( (float*)(ptr), out )
12353+ #define stbir__simdi8_store( ptr, reg ) _mm256_storeu_si256( (__m256i*)(ptr), reg )
12354+ #define stbir__simdf8_frep8( fval ) _mm256_set1_ps( fval )
12355+
12356+ #define stbir__simdf8_min( out, reg0, reg1 ) (out) = _mm256_min_ps( reg0, reg1 )
12357+ #define stbir__simdf8_max( out, reg0, reg1 ) (out) = _mm256_max_ps( reg0, reg1 )
12358+
12359+ #define stbir__simdf8_add4halves( out, bot4, top8 ) (out) = _mm_add_ps( bot4, _mm256_extractf128_ps( top8, 1 ) )
12360+ #define stbir__simdf8_mult_mem( out, reg, ptr ) (out) = _mm256_mul_ps( reg, _mm256_loadu_ps( (float const*)(ptr) ) )
12361+ #define stbir__simdf8_add_mem( out, reg, ptr ) (out) = _mm256_add_ps( reg, _mm256_loadu_ps( (float const*)(ptr) ) )
12362+ #define stbir__simdf8_add( out, a, b ) (out) = _mm256_add_ps( a, b )
12363+ #define stbir__simdf8_load1b( out, ptr ) (out) = _mm256_broadcast_ss( ptr )
12364+ #define stbir__simdf_load1rep4( out, ptr ) (out) = _mm_broadcast_ss( ptr ) // avx load instruction
12365+
12366+ #define stbir__simdi8_convert_i32_to_float(out, ireg) (out) = _mm256_cvtepi32_ps( ireg )
12367+ #define stbir__simdf8_convert_float_to_i32( i, f ) (i) = _mm256_cvttps_epi32(f)
12368+
12369+ #define stbir__simdf8_bot4s( out, a, b ) (out) = _mm256_permute2f128_ps(a,b, (0<<0)+(2<<4) )
12370+ #define stbir__simdf8_top4s( out, a, b ) (out) = _mm256_permute2f128_ps(a,b, (1<<0)+(3<<4) )
12371+
12372+ #define stbir__simdf8_gettop4( reg ) _mm256_extractf128_ps(reg,1)
12373+
12374+ #ifdef STBIR_AVX2
12375+
12376+ // vpmovzxbd takes eight bytes to eight dwords in one shuffle, so sixteen bytes are three (the high eight moved down first)
12377+ #define stbir__simdi8_expand_u8_to_u32(out0,out1,ireg) \
12378+ { \
12379+ out0 = _mm256_cvtepu8_epi32( ireg ); \
12380+ out1 = _mm256_cvtepu8_epi32( _mm_unpackhi_epi64( ireg, ireg ) ); \
12381+ }
12382+
12383+ #define stbir__simdf8_pack_to_16bytes(out,aa,bb) \
12384+ { \
12385+ stbir__simdi8 t; \
12386+ stbir__simdf8 af,bf; \
12387+ stbir__simdi8 a,b; \
12388+ af = _mm256_min_ps( aa, STBIR_max_uint8_as_float8 ); \
12389+ bf = _mm256_min_ps( bb, STBIR_max_uint8_as_float8 ); \
12390+ af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
12391+ bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
12392+ a = _mm256_cvttps_epi32( af ); \
12393+ b = _mm256_cvttps_epi32( bf ); \
12394+ t = _mm256_permute4x64_epi64( _mm256_packs_epi32( a, b ), (0<<0)+(2<<2)+(1<<4)+(3<<6) ); \
12395+ out = _mm256_castsi256_si128( _mm256_permute4x64_epi64( _mm256_packus_epi16( t, t ), (0<<0)+(2<<2)+(1<<4)+(3<<6) ) ); \
12396+ }
12397+
12398+ #define stbir__simdi8_expand_u16_to_u32(out,ireg) out = _mm256_cvtepu16_epi32( ireg );
12399+
12400+ // a byte permute of 32 bytes by one 16 byte table, per 128-bit lane (vpshufb ymm)
12401+ #define stbir__simdi8_shuffle_bytes( out, reg, table ) (out) = _mm256_shuffle_epi8( (reg), _mm256_broadcastsi128_si256( table ) )
12402+
12403+ #define stbir__simdf8_pack_to_16words(out,aa,bb) \
12404+ { \
12405+ stbir__simdf8 af,bf; \
12406+ stbir__simdi8 a,b; \
12407+ af = _mm256_min_ps( aa, STBIR_max_uint16_as_float8 ); \
12408+ bf = _mm256_min_ps( bb, STBIR_max_uint16_as_float8 ); \
12409+ af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
12410+ bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
12411+ a = _mm256_cvttps_epi32( af ); \
12412+ b = _mm256_cvttps_epi32( bf ); \
12413+ (out) = _mm256_permute4x64_epi64( _mm256_packus_epi32(a, b), (0<<0)+(2<<2)+(1<<4)+(3<<6) ); \
12414+ }
12415+
12416+ #else
12417+
12418+ #define stbir__simdi8_expand_u8_to_u32(out0,out1,ireg) \
12419+ { \
12420+ stbir__simdi a,zero = _mm_setzero_si128(); \
12421+ a = _mm_unpacklo_epi8( ireg, zero ); \
12422+ out0 = _mm256_setr_m128i( _mm_unpacklo_epi16( a, zero ), _mm_unpackhi_epi16( a, zero ) ); \
12423+ a = _mm_unpackhi_epi8( ireg, zero ); \
12424+ out1 = _mm256_setr_m128i( _mm_unpacklo_epi16( a, zero ), _mm_unpackhi_epi16( a, zero ) ); \
12425+ }
12426+
12427+ #define stbir__simdf8_pack_to_16bytes(out,aa,bb) \
12428+ { \
12429+ stbir__simdi t; \
12430+ stbir__simdf8 af,bf; \
12431+ stbir__simdi8 a,b; \
12432+ af = _mm256_min_ps( aa, STBIR_max_uint8_as_float8 ); \
12433+ bf = _mm256_min_ps( bb, STBIR_max_uint8_as_float8 ); \
12434+ af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
12435+ bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
12436+ a = _mm256_cvttps_epi32( af ); \
12437+ b = _mm256_cvttps_epi32( bf ); \
12438+ out = _mm_packs_epi32( _mm256_castsi256_si128(a), _mm256_extractf128_si256( a, 1 ) ); \
12439+ out = _mm_packus_epi16( out, out ); \
12440+ t = _mm_packs_epi32( _mm256_castsi256_si128(b), _mm256_extractf128_si256( b, 1 ) ); \
12441+ t = _mm_packus_epi16( t, t ); \
12442+ out = _mm_castps_si128( _mm_shuffle_ps( _mm_castsi128_ps(out), _mm_castsi128_ps(t), (0<<0)+(1<<2)+(0<<4)+(1<<6) ) ); \
12443+ }
12444+
12445+ #define stbir__simdi8_expand_u16_to_u32(out,ireg) \
12446+ { \
12447+ stbir__simdi a,b,zero = _mm_setzero_si128(); \
12448+ a = _mm_unpacklo_epi16( ireg, zero ); \
12449+ b = _mm_unpackhi_epi16( ireg, zero ); \
12450+ out = _mm256_insertf128_si256( _mm256_castsi128_si256( a ), b, 1 ); \
12451+ }
12452+
12453+ #define stbir__simdf8_pack_to_16words(out,aa,bb) \
12454+ { \
12455+ stbir__simdi t0,t1; \
12456+ stbir__simdf8 af,bf; \
12457+ stbir__simdi8 a,b; \
12458+ af = _mm256_min_ps( aa, STBIR_max_uint16_as_float8 ); \
12459+ bf = _mm256_min_ps( bb, STBIR_max_uint16_as_float8 ); \
12460+ af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
12461+ bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
12462+ a = _mm256_cvttps_epi32( af ); \
12463+ b = _mm256_cvttps_epi32( bf ); \
12464+ t0 = _mm_packus_epi32( _mm256_castsi256_si128(a), _mm256_extractf128_si256( a, 1 ) ); \
12465+ t1 = _mm_packus_epi32( _mm256_castsi256_si128(b), _mm256_extractf128_si256( b, 1 ) ); \
12466+ out = _mm256_setr_m128i( t0, t1 ); \
12467+ }
12468+
12469+ #endif
12470+
12471+ static __m256i stbir_00001111 = { STBIR__CONST_4d_32i( 0, 0, 0, 0 ), STBIR__CONST_4d_32i( 1, 1, 1, 1 ) };
12472+ #define stbir__simdf8_0123to00001111( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_00001111 )
12473+
12474+ static __m256i stbir_22223333 = { STBIR__CONST_4d_32i( 2, 2, 2, 2 ), STBIR__CONST_4d_32i( 3, 3, 3, 3 ) };
12475+ #define stbir__simdf8_0123to22223333( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_22223333 )
12476+
12477+ #define stbir__simdf8_0123to2222( out, in ) (out) = stbir__simdf_swiz(_mm256_castps256_ps128(in), 2,2,2,2 )
12478+
12479+ #define stbir__simdf8_load4b( out, ptr ) (out) = _mm256_broadcast_ps( (__m128 const *)(ptr) )
12480+
12481+ static __m256i stbir_00112233 = { STBIR__CONST_4d_32i( 0, 0, 1, 1 ), STBIR__CONST_4d_32i( 2, 2, 3, 3 ) };
12482+ #define stbir__simdf8_0123to00112233( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_00112233 )
12483+ #define stbir__simdf8_add4( out, a8, b ) (out) = _mm256_add_ps( a8, _mm256_insertf128_ps( _mm256_setzero_ps(), b, 0 ) )
12484+
12485+ static __m256i stbir_load6 = { STBIR__CONST_4_32i( 0x80000000 ), STBIR__CONST_4d_32i( 0x80000000, 0x80000000, 0, 0 ) };
12486+ #define stbir__simdf8_load6z( out, ptr ) (out) = _mm256_maskload_ps( ptr, stbir_load6 )
12487+
12488+ #define stbir__simdf8_0123to00000000( out, in ) (out) = _mm256_shuffle_ps ( in, in, (0<<0)+(0<<2)+(0<<4)+(0<<6) )
12489+ #define stbir__simdf8_0123to11111111( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(1<<2)+(1<<4)+(1<<6) )
12490+ #define stbir__simdf8_0123to22222222( out, in ) (out) = _mm256_shuffle_ps ( in, in, (2<<0)+(2<<2)+(2<<4)+(2<<6) )
12491+ #define stbir__simdf8_0123to33333333( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(3<<2)+(3<<4)+(3<<6) )
12492+ #define stbir__simdf8_0123to21032103( out, in ) (out) = _mm256_shuffle_ps ( in, in, (2<<0)+(1<<2)+(0<<4)+(3<<6) )
12493+ #define stbir__simdf8_0123to32103210( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(2<<2)+(1<<4)+(0<<6) )
12494+ #define stbir__simdf8_0123to12301230( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(2<<2)+(3<<4)+(0<<6) )
12495+ #define stbir__simdf8_0123to10321032( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(0<<2)+(3<<4)+(2<<6) )
12496+ #define stbir__simdf8_0123to30123012( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(0<<2)+(1<<4)+(2<<6) )
12497+
12498+ #define stbir__simdf8_0123to11331133( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(1<<2)+(3<<4)+(3<<6) )
12499+ #define stbir__simdf8_0123to00220022( out, in ) (out) = _mm256_shuffle_ps ( in, in, (0<<0)+(0<<2)+(2<<4)+(2<<6) )
12500+
12501+ #define stbir__simdf8_aaa1( out, alp, ones ) do { (out) = _mm256_blend_ps( alp, ones, (1<<0)+(1<<1)+(1<<2)+(0<<3)+(1<<4)+(1<<5)+(1<<6)+(0<<7)); (out)=_mm256_shuffle_ps( out,out, (3<<0) + (3<<2) + (3<<4) + (0<<6) ); } while(0)
12502+ #define stbir__simdf8_1aaa( out, alp, ones ) do { (out) = _mm256_blend_ps( alp, ones, (0<<0)+(1<<1)+(1<<2)+(1<<3)+(0<<4)+(1<<5)+(1<<6)+(1<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (0<<4) + (0<<6) ); } while(0)
12503+ #define stbir__simdf8_a1a1( out, alp, ones) do { (out) = _mm256_blend_ps( alp, ones, (1<<0)+(0<<1)+(1<<2)+(0<<3)+(1<<4)+(0<<5)+(1<<6)+(0<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (3<<4) + (2<<6) ); } while(0)
12504+ #define stbir__simdf8_1a1a( out, alp, ones) do { (out) = _mm256_blend_ps( alp, ones, (0<<0)+(1<<1)+(0<<2)+(1<<3)+(0<<4)+(1<<5)+(0<<6)+(1<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (3<<4) + (2<<6) ); } while(0)
12505+
12506+ #define stbir__simdf8_zero( reg ) (reg) = _mm256_setzero_ps()
12507+
12508+ #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd
12509+ #define stbir__simdf8_madd( out, add, mul1, mul2 ) (out) = _mm256_fmadd_ps( mul1, mul2, add )
12510+ #define stbir__simdf8_madd_mem( out, add, mul, ptr ) (out) = _mm256_fmadd_ps( mul, _mm256_loadu_ps( (float const*)(ptr) ), add )
12511+ #define stbir__simdf8_madd_mem4( out, add, mul, ptr )(out) = _mm256_fmadd_ps( _mm256_setr_m128( mul, _mm_setzero_ps() ), _mm256_setr_m128( _mm_loadu_ps( (float const*)(ptr) ), _mm_setzero_ps() ), add )
12512+ #else
12513+ #define stbir__simdf8_madd( out, add, mul1, mul2 ) (out) = _mm256_add_ps( add, _mm256_mul_ps( mul1, mul2 ) )
12514+ #define stbir__simdf8_madd_mem( out, add, mul, ptr ) (out) = _mm256_add_ps( add, _mm256_mul_ps( mul, _mm256_loadu_ps( (float const*)(ptr) ) ) )
12515+ #define stbir__simdf8_madd_mem4( out, add, mul, ptr ) (out) = _mm256_add_ps( add, _mm256_setr_m128( _mm_mul_ps( mul, _mm_loadu_ps( (float const*)(ptr) ) ), _mm_setzero_ps() ) )
12516+ #endif
12517+ #define stbir__if_simdf8_cast_to_simdf4( val ) _mm256_castps256_ps128( val )
12518+
12519+ static const stbir__simdf8 STBIR_max_uint16_as_float_inverted8 = { stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted };
12520+ static const stbir__simdf8 STBIR_max_uint8_as_float_inverted8 = { stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted };
12521+ static const stbir__simdf8 STBIR_ones8 = { 1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0 };
12522+ static const stbir__simdf8 STBIR_simd_point58 = { 0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5 };
12523+ static const stbir__simdf8 STBIR_max_uint8_as_float8 = { stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float, stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float };
12524+ static const stbir__simdf8 STBIR_max_uint16_as_float8 = { stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float, stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float };
12525+
12526+ // the 16-wide set (AVX-512) for the X-wide code.
12527+ #ifdef STBIR_AVX512
12528+ #define STBIR_SIMD16
12529+ #define stbir__simdf16 __m512
12530+ #define stbir__simdi16 __m512i
12531+ #define stbir__simdf16_load( out, ptr ) (out) = _mm512_loadu_ps( (float const *)(ptr) )
12532+ #define stbir__simdi16_load( out, ptr ) (out) = _mm512_loadu_si512( (void const *)(ptr) )
12533+ #define stbir__simdf16_store( ptr, reg ) _mm512_storeu_ps( (float*)(ptr), reg )
12534+ #define stbir__simdi16_store( ptr, reg ) _mm512_storeu_si512( (void*)(ptr), reg )
12535+ #define stbir__simdf16_mult( out, a, b ) (out) = _mm512_mul_ps( (a), (b) )
12536+ #define stbir__simdf16_add_mem( out, reg, ptr ) (out) = _mm512_add_ps( reg, _mm512_loadu_ps( (float const*)(ptr) ) )
12537+ #define stbir__simdf16_min( out, a, b ) (out) = _mm512_min_ps( a, b )
12538+ #define stbir__simdf16_max( out, a, b ) (out) = _mm512_max_ps( a, b )
12539+ #define stbir__simdf16_zero( reg ) (reg) = _mm512_setzero_ps()
12540+ #define stbir__simdf16_frep16( fval ) _mm512_set1_ps( fval )
12541+ #define stbir__simdi16_convert_i32_to_float( out, ireg ) (out) = _mm512_cvtepi32_ps( ireg )
12542+ #define stbir__simdf16_convert_float_to_i32( i, f ) (i) = _mm512_cvttps_epi32( f )
12543+ #define stbir__simdf16_cast_to_simdf4( val ) _mm512_castps512_ps128( val )
12544+
12545+ #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd
12546+ #define stbir__simdf16_madd( out, add, mul1, mul2 ) (out) = _mm512_fmadd_ps( mul1, mul2, add )
12547+ #define stbir__simdf16_madd_mem( out, add, mul, ptr ) (out) = _mm512_fmadd_ps( mul, _mm512_loadu_ps( (float const*)(ptr) ), add )
12548+ #else
12549+ #define stbir__simdf16_madd( out, add, mul1, mul2 ) (out) = _mm512_add_ps( add, _mm512_mul_ps( mul1, mul2 ) )
12550+ #define stbir__simdf16_madd_mem( out, add, mul, ptr ) (out) = _mm512_add_ps( add, _mm512_mul_ps( mul, _mm512_loadu_ps( (float const*)(ptr) ) ) )
12551+ #endif
12552+
12553+ // the per lane shuffles: one 4-wide order, applied in every lane
12554+ #define stbir__simdf16_0123to0123( out, in ) (out) = (in)
12555+ #define stbir__simdf16_0123to1230( out, in ) (out) = _mm512_shuffle_ps( in, in, (1<<0)+(2<<2)+(3<<4)+(0<<6) )
12556+ #define stbir__simdf16_0123to2103( out, in ) (out) = _mm512_shuffle_ps( in, in, (2<<0)+(1<<2)+(0<<4)+(3<<6) )
12557+ #define stbir__simdf16_0123to3210( out, in ) (out) = _mm512_shuffle_ps( in, in, (3<<0)+(2<<2)+(1<<4)+(0<<6) )
12558+ #define stbir__simdf16_0123to1032( out, in ) (out) = _mm512_shuffle_ps( in, in, (1<<0)+(0<<2)+(3<<4)+(2<<6) )
12559+ #define stbir__simdf16_0123to3012( out, in ) (out) = _mm512_shuffle_ps( in, in, (3<<0)+(0<<2)+(1<<4)+(2<<6) )
12560+
12561+ // the alpha patterns: blend the ones in (the mask repeats per lane), then shuffle each lane
12562+ #define stbir__simdf16_aaa1( out, alp, ones ) do { (out) = _mm512_mask_blend_ps( (__mmask16)0x7777, alp, ones ); (out) = _mm512_shuffle_ps( out, out, (3<<0)+(3<<2)+(3<<4)+(0<<6) ); } while(0)
12563+ #define stbir__simdf16_1aaa( out, alp, ones ) do { (out) = _mm512_mask_blend_ps( (__mmask16)0xeeee, alp, ones ); (out) = _mm512_shuffle_ps( out, out, (1<<0)+(0<<2)+(0<<4)+(0<<6) ); } while(0)
12564+ #define stbir__simdf16_a1a1( out, alp, ones ) do { (out) = _mm512_mask_blend_ps( (__mmask16)0x5555, alp, ones ); (out) = _mm512_shuffle_ps( out, out, (1<<0)+(0<<2)+(3<<4)+(2<<6) ); } while(0)
12565+ #define stbir__simdf16_1a1a( out, alp, ones ) do { (out) = _mm512_mask_blend_ps( (__mmask16)0xaaaa, alp, ones ); (out) = _mm512_shuffle_ps( out, out, (1<<0)+(0<<2)+(3<<4)+(2<<6) ); } while(0)
12566+
12567+ // integer conversions: sixteen bytes or words to sixteen dwords in one op, and the saturating narrows back
12568+ #define stbir__simdi16_expand_u8_to_u32( out, ireg ) (out) = _mm512_cvtepu8_epi32( ireg )
12569+ #define stbir__simdi16_expand_u16_to_u32( out, ireg ) (out) = _mm512_cvtepu16_epi32( ireg )
12570+
12571+ #define stbir__simdf16_pack_to_32bytes( out, aa, bb ) \
12572+ { \
12573+ stbir__simdf16 af, bf; \
12574+ af = _mm512_min_ps( aa, STBIR_max_uint8_as_float16 ); \
12575+ bf = _mm512_min_ps( bb, STBIR_max_uint8_as_float16 ); \
12576+ af = _mm512_max_ps( af, _mm512_setzero_ps() ); \
12577+ bf = _mm512_max_ps( bf, _mm512_setzero_ps() ); \
12578+ (out) = _mm256_inserti128_si256( _mm256_castsi128_si256( _mm512_cvtusepi32_epi8( _mm512_cvttps_epi32( af ) ) ), _mm512_cvtusepi32_epi8( _mm512_cvttps_epi32( bf ) ), 1 ); \
12579+ }
12580+
12581+ // the pack interleaves the two inputs per 128-bit lane, the 64-bit permute puts them back in order
12582+ static const stbir__simdi16 stbir__pack_lanes_in_order16 = { STBIR__CONST_4d_32i( 0,0, 2,0 ), STBIR__CONST_4d_32i( 4,0, 6,0 ), STBIR__CONST_4d_32i( 1,0, 3,0 ), STBIR__CONST_4d_32i( 5,0, 7,0 ) }; // 64-bit indexes 0 2 4 6 1 3 5 7
12583+ #define stbir__simdf16_pack_to_32words( out, aa, bb ) \
12584+ { \
12585+ stbir__simdf16 af, bf; \
12586+ af = _mm512_min_ps( aa, STBIR_max_uint16_as_float16 ); \
12587+ bf = _mm512_min_ps( bb, STBIR_max_uint16_as_float16 ); \
12588+ af = _mm512_max_ps( af, _mm512_setzero_ps() ); \
12589+ bf = _mm512_max_ps( bf, _mm512_setzero_ps() ); \
12590+ (out) = _mm512_permutexvar_epi64( stbir__pack_lanes_in_order16, _mm512_packus_epi32( _mm512_cvttps_epi32( af ), _mm512_cvttps_epi32( bf ) ) ); \
12591+ }
12592+
12593+ static const stbir__simdf16 STBIR_ones16 = { 1.0f,1.0f,1.0f,1.0f, 1.0f,1.0f,1.0f,1.0f, 1.0f,1.0f,1.0f,1.0f, 1.0f,1.0f,1.0f,1.0f };
12594+ static const stbir__simdf16 STBIR_simd_point516 = { 0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f };
12595+ static const stbir__simdf16 STBIR_max_uint8_as_float16 = { 255.0f,255.0f,255.0f,255.0f, 255.0f,255.0f,255.0f,255.0f, 255.0f,255.0f,255.0f,255.0f, 255.0f,255.0f,255.0f,255.0f };
12596+ static const stbir__simdf16 STBIR_max_uint16_as_float16 = { 65535.0f,65535.0f,65535.0f,65535.0f, 65535.0f,65535.0f,65535.0f,65535.0f, 65535.0f,65535.0f,65535.0f,65535.0f, 65535.0f,65535.0f,65535.0f,65535.0f };
12597+ static const stbir__simdf16 STBIR_max_uint8_as_float_inverted16 = { stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted, stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted, stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted, stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted };
12598+ static const stbir__simdf16 STBIR_max_uint16_as_float_inverted16 = { stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted, stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted, stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted, stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted };
12599+ #endif // STBIR_AVX512
12600+
12601+ #endif
12602+
12603+#elif defined(STBIR_NEON)
12604+
12605+ #include <arm_neon.h>
12606+
12607+ #define stbir__simdf float32x4_t
12608+ #define stbir__simdi uint32x4_t
12609+
12610+ #define stbir_simdi_castf( reg ) vreinterpretq_u32_f32(reg)
12611+ #define stbir_simdf_casti( reg ) vreinterpretq_f32_u32(reg)
12612+
12613+ #define stbir__simdf_load( reg, ptr ) (reg) = vld1q_f32( (float const*)(ptr) )
12614+ #define stbir__simdi_load( reg, ptr ) (reg) = vld1q_u32( (uint32_t const*)(ptr) )
12615+ #define stbir__simdf_load1( out, ptr ) (out) = vld1q_lane_f32( (float const*)(ptr), vdupq_n_f32(0), 0 ) // top values zero, like SSE (a replicated inf would turn into nan against the zero coefficient lanes)
12616+ #define stbir__simdf_load1z( out, ptr ) (out) = vld1q_lane_f32( (float const*)(ptr), vdupq_n_f32(0), 0 ) // top values must be zero
12617+ #define stbir__simdf_frep4( fvar ) vdupq_n_f32( fvar )
12618+ #define stbir__simdf_load1frep4( out, fvar ) (out) = vdupq_n_f32( fvar )
12619+ #define stbir__simdf_load2( out, ptr ) (out) = vcombine_f32( vld1_f32( (float const*)(ptr) ), vcreate_f32(0) ) // top values can be random (not denormal or nan for perf)
12620+ #define stbir__simdf_load2z( out, ptr ) (out) = vcombine_f32( vld1_f32( (float const*)(ptr) ), vcreate_f32(0) ) // top values must be zero
12621+ #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = vcombine_f32( vget_low_f32(reg), vld1_f32( (float const*)(ptr) ) )
12622+ // pairwise add of two registers: out = ( a0+a1, a2+a3, b0+b1, b2+b3 ), and the halves version: ( a0+a2, a1+a3, b0+b2, b1+b3 )
12623+ #define stbir__simdf_hadd2( out, a, b ) (out) = vcombine_f32( vpadd_f32( vget_low_f32( a ), vget_high_f32( a ) ), vpadd_f32( vget_low_f32( b ), vget_high_f32( b ) ) )
12624+ #define stbir__simdf_addhalves2( out, a, b ) (out) = vaddq_f32( vcombine_f32( vget_low_f32( a ), vget_low_f32( b ) ), vcombine_f32( vget_high_f32( a ), vget_high_f32( b ) ) )
12625+
12626+ #define stbir__simdf_zeroP() vdupq_n_f32(0)
12627+ #define stbir__simdf_zero( reg ) (reg) = vdupq_n_f32(0)
12628+
12629+ #define stbir__simdf_store( ptr, reg ) vst1q_f32( (float*)(ptr), reg )
12630+ #define stbir__simdf_store1( ptr, reg ) vst1q_lane_f32( (float*)(ptr), reg, 0)
12631+ #define stbir__simdf_store2( ptr, reg ) vst1_f32( (float*)(ptr), vget_low_f32(reg) )
12632+ #define stbir__simdf_store2h( ptr, reg ) vst1_f32( (float*)(ptr), vget_high_f32(reg) )
12633+
12634+ #define stbir__simdi_store( ptr, reg ) vst1q_u32( (uint32_t*)(ptr), reg )
12635+ #define stbir__simdi_store2( ptr, reg ) vst1_u32( (uint32_t*)(ptr), vget_low_u32(reg) )
12636+
12637+ #define stbir__prefetch( ptr )
12638+
12639+ #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
12640+ { \
12641+ uint16x8_t l = vmovl_u8( vget_low_u8 ( vreinterpretq_u8_u32(ireg) ) ); \
12642+ uint16x8_t h = vmovl_u8( vget_high_u8( vreinterpretq_u8_u32(ireg) ) ); \
12643+ out0 = vmovl_u16( vget_low_u16 ( l ) ); \
12644+ out1 = vmovl_u16( vget_high_u16( l ) ); \
12645+ out2 = vmovl_u16( vget_low_u16 ( h ) ); \
12646+ out3 = vmovl_u16( vget_high_u16( h ) ); \
12647+ }
12648+
12649+
12650+ #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
12651+ { \
12652+ uint16x8_t tmp = vreinterpretq_u16_u32(ireg); \
12653+ out0 = vmovl_u16( vget_low_u16 ( tmp ) ); \
12654+ out1 = vmovl_u16( vget_high_u16( tmp ) ); \
12655+ }
12656+
12657+ #define stbir__simdf_convert_float_to_i32( i, f ) (i) = vreinterpretq_u32_s32( vcvtq_s32_f32(f) )
12658+ #define stbir__simdi_to_int( i ) (int)vgetq_lane_u32(i, 0)
12659+ #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)vgetq_lane_s32(vcvtq_s32_f32(vmaxq_f32(vminq_f32(f,STBIR__CONSTF(STBIR_max_uint8_as_float)),vdupq_n_f32(0))), 0))
12660+ #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)vgetq_lane_s32(vcvtq_s32_f32(vmaxq_f32(vminq_f32(f,STBIR__CONSTF(STBIR_max_uint16_as_float)),vdupq_n_f32(0))), 0))
12661+ #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = vcvtq_f32_s32( vreinterpretq_s32_u32(ireg) )
12662+ #define stbir__simdf_add( out, reg0, reg1 ) (out) = vaddq_f32( reg0, reg1 )
12663+ #define stbir__simdf_mult( out, reg0, reg1 ) (out) = vmulq_f32( reg0, reg1 )
12664+ #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = vmulq_f32( reg, vld1q_f32( (float const*)(ptr) ) )
12665+ #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = vmulq_f32( reg, vld1q_dup_f32( (float const*)(ptr) ) )
12666+ #define stbir__simdf_add_mem( out, reg, ptr ) (out) = vaddq_f32( reg, vld1q_f32( (float const*)(ptr) ) )
12667+
12668+ #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd (and also x64 no madd to arm madd)
12669+ #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = vfmaq_f32( add, mul1, mul2 )
12670+ #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = vfmaq_f32( add, mul, vld1q_f32( (float const*)(ptr) ) )
12671+ #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = vfmaq_f32( add, mul, vld1q_dup_f32( (float const*)(ptr) ) )
12672+ #else
12673+ #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = vaddq_f32( add, vmulq_f32( mul1, mul2 ) )
12674+ #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = vaddq_f32( add, vmulq_f32( mul, vld1q_f32( (float const*)(ptr) ) ) )
12675+ #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = vaddq_f32( add, vmulq_f32( mul, vld1q_dup_f32( (float const*)(ptr) ) ) )
12676+ #endif
12677+
12678+ #define stbir__simdf_add1( out, reg0, reg1 ) (out) = vaddq_f32( reg0, reg1 )
12679+
12680+ #define stbir__simdf_and( out, reg0, reg1 ) (out) = vreinterpretq_f32_u32( vandq_u32( vreinterpretq_u32_f32(reg0), vreinterpretq_u32_f32(reg1) ) )
12681+
12682+ // min / max with SSE's nan rule
12683+ #if defined( STBIR__ARM64 )
12684+ #define stbir__simdf_min( out, reg0, reg1 ) (out) = vminnmq_f32( reg0, reg1 )
12685+ #define stbir__simdf_max( out, reg0, reg1 ) (out) = vmaxnmq_f32( reg0, reg1 )
12686+ #else
12687+ #define stbir__simdf_min( out, reg0, reg1 ) (out) = vbslq_f32( vcltq_f32( reg0, reg1 ), reg0, reg1 )
12688+ #define stbir__simdf_max( out, reg0, reg1 ) (out) = vbslq_f32( vcgtq_f32( reg0, reg1 ), reg0, reg1 )
12689+ #endif
12690+
12691+ #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out) = vextq_f32( reg0, reg1, 3 )
12692+ #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out) = vextq_f32( reg0, reg1, 2 )
12693+
12694+ #define stbir__simdf_a1a1( out, alp, ones ) (out) = vzipq_f32(vuzpq_f32(alp, alp).val[1], ones).val[0]
12695+ #define stbir__simdf_1a1a( out, alp, ones ) (out) = vzipq_f32(ones, vuzpq_f32(alp, alp).val[0]).val[0]
12696+
12697+ #if defined( STBIR__ARM64 )
12698+
12699+ #define stbir__simdf_aaa1( out, alp, ones ) (out) = vcopyq_laneq_f32(vdupq_n_f32(vgetq_lane_f32(alp, 3)), 3, ones, 3)
12700+ #define stbir__simdf_1aaa( out, alp, ones ) (out) = vcopyq_laneq_f32(vdupq_n_f32(vgetq_lane_f32(alp, 0)), 0, ones, 0)
12701+
12702+ #if defined( _MSC_VER ) && !defined(__clang__)
12703+ #define stbir_make16(a,b,c,d) vcombine_u8( \
12704+ vcreate_u8( (4*a+0) | ((4*a+1)<<8) | ((4*a+2)<<16) | ((4*a+3)<<24) | \
12705+ ((stbir_uint64)(4*b+0)<<32) | ((stbir_uint64)(4*b+1)<<40) | ((stbir_uint64)(4*b+2)<<48) | ((stbir_uint64)(4*b+3)<<56)), \
12706+ vcreate_u8( (4*c+0) | ((4*c+1)<<8) | ((4*c+2)<<16) | ((4*c+3)<<24) | \
12707+ ((stbir_uint64)(4*d+0)<<32) | ((stbir_uint64)(4*d+1)<<40) | ((stbir_uint64)(4*d+2)<<48) | ((stbir_uint64)(4*d+3)<<56) ) )
12708+
12709+ static stbir__inline uint8x16x2_t stbir_make16x2(float32x4_t rega,float32x4_t regb)
12710+ {
12711+ uint8x16x2_t r = { vreinterpretq_u8_f32(rega), vreinterpretq_u8_f32(regb) };
12712+ return r;
12713+ }
12714+ #else
12715+ #define stbir_make16(a,b,c,d) (uint8x16_t){4*a+0,4*a+1,4*a+2,4*a+3,4*b+0,4*b+1,4*b+2,4*b+3,4*c+0,4*c+1,4*c+2,4*c+3,4*d+0,4*d+1,4*d+2,4*d+3}
12716+ #define stbir_make16x2(a,b) (uint8x16x2_t){{vreinterpretq_u8_f32(a),vreinterpretq_u8_f32(b)}}
12717+ #endif
12718+
12719+ #define stbir__simdf_swiz( reg, one, two, three, four ) vreinterpretq_f32_u8( vqtbl1q_u8( vreinterpretq_u8_f32(reg), stbir_make16(one, two, three, four) ) )
12720+ #define stbir__simdf_swiz2( rega, regb, one, two, three, four ) vreinterpretq_f32_u8( vqtbl2q_u8( stbir_make16x2(rega,regb), stbir_make16(one, two, three, four) ) )
12721+
12722+ // an arbitrary byte permute
12723+ #define STBIR_SIMD_BYTE_SHUFFLE
12724+ #define stbir__simdi_shuffle_bytes( out, reg, table ) (out) = vreinterpretq_u32_u8( vqtbl1q_u8( vreinterpretq_u8_u32( reg ), vreinterpretq_u8_u32( table ) ) )
12725+
12726+ #define stbir__simdi_16madd( out, reg0, reg1 ) \
12727+ { \
12728+ int16x8_t r0 = vreinterpretq_s16_u32(reg0); \
12729+ int16x8_t r1 = vreinterpretq_s16_u32(reg1); \
12730+ int32x4_t tmp0 = vmull_s16( vget_low_s16(r0), vget_low_s16(r1) ); \
12731+ int32x4_t tmp1 = vmull_s16( vget_high_s16(r0), vget_high_s16(r1) ); \
12732+ (out) = vreinterpretq_u32_s32( vpaddq_s32(tmp0, tmp1) ); \
12733+ }
12734+
12735+ #else
12736+
12737+ #define stbir__simdf_aaa1( out, alp, ones ) (out) = vsetq_lane_f32(1.0f, vdupq_n_f32(vgetq_lane_f32(alp, 3)), 3)
12738+ #define stbir__simdf_1aaa( out, alp, ones ) (out) = vsetq_lane_f32(1.0f, vdupq_n_f32(vgetq_lane_f32(alp, 0)), 0)
12739+
12740+ #if defined( _MSC_VER ) && !defined(__clang__)
12741+ static stbir__inline uint8x8x2_t stbir_make8x2(float32x4_t reg)
12742+ {
12743+ uint8x8x2_t r = { { vget_low_u8(vreinterpretq_u8_f32(reg)), vget_high_u8(vreinterpretq_u8_f32(reg)) } };
12744+ return r;
12745+ }
12746+ #define stbir_make8(a,b) vcreate_u8( \
12747+ (4*a+0) | ((4*a+1)<<8) | ((4*a+2)<<16) | ((4*a+3)<<24) | \
12748+ ((stbir_uint64)(4*b+0)<<32) | ((stbir_uint64)(4*b+1)<<40) | ((stbir_uint64)(4*b+2)<<48) | ((stbir_uint64)(4*b+3)<<56) )
12749+ #else
12750+ #define stbir_make8x2(reg) (uint8x8x2_t){ { vget_low_u8(vreinterpretq_u8_f32(reg)), vget_high_u8(vreinterpretq_u8_f32(reg)) } }
12751+ #define stbir_make8(a,b) (uint8x8_t){4*a+0,4*a+1,4*a+2,4*a+3,4*b+0,4*b+1,4*b+2,4*b+3}
12752+ #endif
12753+
12754+ #define stbir__simdf_swiz( reg, one, two, three, four ) vreinterpretq_f32_u8( vcombine_u8( \
12755+ vtbl2_u8( stbir_make8x2( reg ), stbir_make8( one, two ) ), \
12756+ vtbl2_u8( stbir_make8x2( reg ), stbir_make8( three, four ) ) ) )
12757+
12758+ #define stbir__simdi_16madd( out, reg0, reg1 ) \
12759+ { \
12760+ int16x8_t r0 = vreinterpretq_s16_u32(reg0); \
12761+ int16x8_t r1 = vreinterpretq_s16_u32(reg1); \
12762+ int32x4_t tmp0 = vmull_s16( vget_low_s16(r0), vget_low_s16(r1) ); \
12763+ int32x4_t tmp1 = vmull_s16( vget_high_s16(r0), vget_high_s16(r1) ); \
12764+ int32x2_t out0 = vpadd_s32( vget_low_s32(tmp0), vget_high_s32(tmp0) ); \
12765+ int32x2_t out1 = vpadd_s32( vget_low_s32(tmp1), vget_high_s32(tmp1) ); \
12766+ (out) = vreinterpretq_u32_s32( vcombine_s32(out0, out1) ); \
12767+ }
12768+
12769+ #endif
12770+
12771+ #define stbir__simdi_and( out, reg0, reg1 ) (out) = vandq_u32( reg0, reg1 )
12772+ #define stbir__simdi_or( out, reg0, reg1 ) (out) = vorrq_u32( reg0, reg1 )
12773+
12774+ #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
12775+ { \
12776+ float32x4_t af = vmaxq_f32( vminq_f32(aa,STBIR__CONSTF(STBIR_max_uint8_as_float) ), vdupq_n_f32(0) ); \
12777+ float32x4_t bf = vmaxq_f32( vminq_f32(bb,STBIR__CONSTF(STBIR_max_uint8_as_float) ), vdupq_n_f32(0) ); \
12778+ int16x4_t ai = vqmovn_s32( vcvtq_s32_f32( af ) ); \
12779+ int16x4_t bi = vqmovn_s32( vcvtq_s32_f32( bf ) ); \
12780+ uint8x8_t out8 = vqmovun_s16( vcombine_s16(ai, bi) ); \
12781+ out = vreinterpretq_u32_u8( vcombine_u8(out8, out8) ); \
12782+ }
12783+
12784+ #define stbir__simdf_pack_to_8words(out,aa,bb) \
12785+ { \
12786+ float32x4_t af = vmaxq_f32( vminq_f32(aa,STBIR__CONSTF(STBIR_max_uint16_as_float) ), vdupq_n_f32(0) ); \
12787+ float32x4_t bf = vmaxq_f32( vminq_f32(bb,STBIR__CONSTF(STBIR_max_uint16_as_float) ), vdupq_n_f32(0) ); \
12788+ int32x4_t ai = vcvtq_s32_f32( af ); \
12789+ int32x4_t bi = vcvtq_s32_f32( bf ); \
12790+ out = vreinterpretq_u32_u16( vcombine_u16(vqmovun_s32(ai), vqmovun_s32(bi)) ); \
12791+ }
12792+
12793+ #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
12794+ { \
12795+ int16x4x2_t tmp0 = vzip_s16( vqmovn_s32(vreinterpretq_s32_u32(r0)), vqmovn_s32(vreinterpretq_s32_u32(r2)) ); \
12796+ int16x4x2_t tmp1 = vzip_s16( vqmovn_s32(vreinterpretq_s32_u32(r1)), vqmovn_s32(vreinterpretq_s32_u32(r3)) ); \
12797+ uint8x8x2_t out = \
12798+ { { \
12799+ vqmovun_s16( vcombine_s16(tmp0.val[0], tmp0.val[1]) ), \
12800+ vqmovun_s16( vcombine_s16(tmp1.val[0], tmp1.val[1]) ), \
12801+ } }; \
12802+ vst2_u8(ptr, out); \
12803+ }
12804+
12805+ #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
12806+ { \
12807+ float32x4x4_t tmp = vld4q_f32(ptr); \
12808+ o0 = tmp.val[0]; \
12809+ o1 = tmp.val[1]; \
12810+ o2 = tmp.val[2]; \
12811+ o3 = tmp.val[3]; \
12812+ }
12813+
12814+ #define stbir__simdi_32shr( out, reg, imm ) out = vshrq_n_u32( reg, imm )
12815+
12816+ #if defined( _MSC_VER ) && !defined(__clang__)
12817+ #define STBIR__SIMDF_CONST(var, x) __declspec(align(8)) float var[] = { x, x, x, x }
12818+ #define STBIR__SIMDI_CONST(var, x) __declspec(align(8)) uint32_t var[] = { x, x, x, x }
12819+ #define STBIR__SIMDI_CONST4(var, a, b, c, d) __declspec(align(8)) uint32_t var[] = { a, b, c, d }
12820+ #define STBIR__CONSTF(var) (*(const float32x4_t*)var)
12821+ #define STBIR__CONSTI(var) (*(const uint32x4_t*)var)
12822+ #else
12823+ #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = { x, x, x, x }
12824+ #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { x, x, x, x }
12825+ #define STBIR__SIMDI_CONST4(var, a, b, c, d) stbir__simdi var = { a, b, c, d }
12826+ #define STBIR__CONSTF(var) (var)
12827+ #define STBIR__CONSTI(var) (var)
12828+ #endif
12829+
12830+ #define STBIR_SIMD
12831+
12832+#elif defined(STBIR_WASM)
12833+
12834+ #include <wasm_simd128.h>
12835+
12836+ #define stbir__simdf v128_t
12837+ #define stbir__simdi v128_t
12838+
12839+ #define stbir_simdi_castf( reg ) (reg)
12840+ #define stbir_simdf_casti( reg ) (reg)
12841+
12842+ #define stbir__simdf_load( reg, ptr ) (reg) = wasm_v128_load( (void const*)(ptr) )
12843+ #define stbir__simdi_load( reg, ptr ) (reg) = wasm_v128_load( (void const*)(ptr) )
12844+ #define stbir__simdf_load1( out, ptr ) (out) = wasm_v128_load32_zero( (void const*)(ptr) ) // top values zero, like SSE (a replicated inf would turn into nan against the zero coefficient lanes)
12845+ #define stbir__simdf_load1z( out, ptr ) (out) = wasm_v128_load32_zero( (void const*)(ptr) ) // top values must be zero
12846+ #define stbir__simdf_frep4( fvar ) wasm_f32x4_splat( fvar )
12847+ #define stbir__simdf_load1frep4( out, fvar ) (out) = wasm_f32x4_splat( fvar )
12848+ #define stbir__simdf_load2( out, ptr ) (out) = wasm_v128_load64_zero( (void const*)(ptr) ) // top values zero, like SSE
12849+ #define stbir__simdf_load2z( out, ptr ) (out) = wasm_v128_load64_zero( (void const*)(ptr) ) // top values must be zero
12850+ #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = wasm_v128_load64_lane( (void const*)(ptr), reg, 1 )
12851+// pairwise add of two registers: out = ( a0+a1, a2+a3, b0+b1, b2+b3 ), and the halves version: ( a0+a2, a1+a3, b0+b2, b1+b3 )
12852+#define stbir__simdf_hadd2( out, a, b ) (out) = wasm_f32x4_add( wasm_i32x4_shuffle( a, b, 0, 2, 4, 6 ), wasm_i32x4_shuffle( a, b, 1, 3, 5, 7 ) )
12853+#define stbir__simdf_addhalves2( out, a, b ) (out) = wasm_f32x4_add( wasm_i32x4_shuffle( a, b, 0, 1, 4, 5 ), wasm_i32x4_shuffle( a, b, 2, 3, 6, 7 ) )
12854+
12855+ #define stbir__simdf_zeroP() wasm_f32x4_const_splat(0)
12856+ #define stbir__simdf_zero( reg ) (reg) = wasm_f32x4_const_splat(0)
12857+
12858+ #define stbir__simdf_store( ptr, reg ) wasm_v128_store( (void*)(ptr), reg )
12859+ #define stbir__simdf_store1( ptr, reg ) wasm_v128_store32_lane( (void*)(ptr), reg, 0 )
12860+ #define stbir__simdf_store2( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 0 )
12861+ #define stbir__simdf_store2h( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 1 )
12862+
12863+ #define stbir__simdi_store( ptr, reg ) wasm_v128_store( (void*)(ptr), reg )
12864+ #define stbir__simdi_store2( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 0 )
12865+
12866+ #define stbir__prefetch( ptr )
12867+
12868+ #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
12869+ { \
12870+ v128_t l = wasm_u16x8_extend_low_u8x16 ( ireg ); \
12871+ v128_t h = wasm_u16x8_extend_high_u8x16( ireg ); \
12872+ out0 = wasm_u32x4_extend_low_u16x8 ( l ); \
12873+ out1 = wasm_u32x4_extend_high_u16x8( l ); \
12874+ out2 = wasm_u32x4_extend_low_u16x8 ( h ); \
12875+ out3 = wasm_u32x4_extend_high_u16x8( h ); \
12876+ }
12877+
12878+
12879+ #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
12880+ { \
12881+ out0 = wasm_u32x4_extend_low_u16x8 ( ireg ); \
12882+ out1 = wasm_u32x4_extend_high_u16x8( ireg ); \
12883+ }
12884+
12885+ #define stbir__simdf_convert_float_to_i32( i, f ) (i) = wasm_i32x4_trunc_sat_f32x4(f)
12886+ #define stbir__simdi_to_int( i ) wasm_i32x4_extract_lane(i, 0)
12887+ #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)wasm_i32x4_extract_lane(wasm_i32x4_trunc_sat_f32x4(wasm_f32x4_max(wasm_f32x4_min(f,STBIR_max_uint8_as_float),wasm_f32x4_const_splat(0))), 0))
12888+ #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)wasm_i32x4_extract_lane(wasm_i32x4_trunc_sat_f32x4(wasm_f32x4_max(wasm_f32x4_min(f,STBIR_max_uint16_as_float),wasm_f32x4_const_splat(0))), 0))
12889+ #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = wasm_f32x4_convert_i32x4(ireg)
12890+ #define stbir__simdf_add( out, reg0, reg1 ) (out) = wasm_f32x4_add( reg0, reg1 )
12891+ #define stbir__simdf_mult( out, reg0, reg1 ) (out) = wasm_f32x4_mul( reg0, reg1 )
12892+ #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = wasm_f32x4_mul( reg, wasm_v128_load( (void const*)(ptr) ) )
12893+ #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = wasm_f32x4_mul( reg, wasm_v128_load32_splat( (void const*)(ptr) ) )
12894+ #define stbir__simdf_add_mem( out, reg, ptr ) (out) = wasm_f32x4_add( reg, wasm_v128_load( (void const*)(ptr) ) )
12895+
12896+ #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul1, mul2 ) )
12897+ #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul, wasm_v128_load( (void const*)(ptr) ) ) )
12898+ #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul, wasm_v128_load32_splat( (void const*)(ptr) ) ) )
12899+
12900+ #define stbir__simdf_add1( out, reg0, reg1 ) (out) = wasm_f32x4_add( reg0, reg1 )
12901+
12902+ #define stbir__simdf_and( out, reg0, reg1 ) (out) = wasm_v128_and( reg0, reg1 )
12903+
12904+ #define stbir__simdf_min( out, reg0, reg1 ) (out) = wasm_f32x4_min( reg0, reg1 )
12905+ #define stbir__simdf_max( out, reg0, reg1 ) (out) = wasm_f32x4_max( reg0, reg1 )
12906+
12907+ #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out) = wasm_i32x4_shuffle( reg0, reg1, 3, 4, 5, -1 )
12908+ #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out) = wasm_i32x4_shuffle( reg0, reg1, 2, 3, 4, -1 )
12909+
12910+ #define stbir__simdf_aaa1(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 3, 3, 3, 4)
12911+ #define stbir__simdf_1aaa(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 4, 0, 0, 0)
12912+ #define stbir__simdf_a1a1(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 1, 4, 3, 4)
12913+ #define stbir__simdf_1a1a(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 4, 0, 4, 2)
12914+
12915+ #define stbir__simdf_swiz( reg, one, two, three, four ) wasm_i32x4_shuffle(reg, reg, one, two, three, four)
12916+
12917+ // an arbitrary byte permute
12918+ #define STBIR_SIMD_BYTE_SHUFFLE
12919+ #define stbir__simdi_shuffle_bytes( out, reg, table ) (out) = wasm_i8x16_swizzle( reg, table )
12920+
12921+ #define stbir__simdi_and( out, reg0, reg1 ) (out) = wasm_v128_and( reg0, reg1 )
12922+ #define stbir__simdi_or( out, reg0, reg1 ) (out) = wasm_v128_or( reg0, reg1 )
12923+ #define stbir__simdi_16madd( out, reg0, reg1 ) (out) = wasm_i32x4_dot_i16x8( reg0, reg1 )
12924+
12925+ #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
12926+ { \
12927+ v128_t af = wasm_f32x4_max( wasm_f32x4_min(aa, STBIR_max_uint8_as_float), wasm_f32x4_const_splat(0) ); \
12928+ v128_t bf = wasm_f32x4_max( wasm_f32x4_min(bb, STBIR_max_uint8_as_float), wasm_f32x4_const_splat(0) ); \
12929+ v128_t ai = wasm_i32x4_trunc_sat_f32x4( af ); \
12930+ v128_t bi = wasm_i32x4_trunc_sat_f32x4( bf ); \
12931+ v128_t out16 = wasm_i16x8_narrow_i32x4( ai, bi ); \
12932+ out = wasm_u8x16_narrow_i16x8( out16, out16 ); \
12933+ }
12934+
12935+ #define stbir__simdf_pack_to_8words(out,aa,bb) \
12936+ { \
12937+ v128_t af = wasm_f32x4_max( wasm_f32x4_min(aa, STBIR_max_uint16_as_float), wasm_f32x4_const_splat(0)); \
12938+ v128_t bf = wasm_f32x4_max( wasm_f32x4_min(bb, STBIR_max_uint16_as_float), wasm_f32x4_const_splat(0)); \
12939+ v128_t ai = wasm_i32x4_trunc_sat_f32x4( af ); \
12940+ v128_t bi = wasm_i32x4_trunc_sat_f32x4( bf ); \
12941+ out = wasm_u16x8_narrow_i32x4( ai, bi ); \
12942+ }
12943+
12944+ #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
12945+ { \
12946+ v128_t tmp0 = wasm_i16x8_narrow_i32x4(r0, r1); \
12947+ v128_t tmp1 = wasm_i16x8_narrow_i32x4(r2, r3); \
12948+ v128_t tmp = wasm_u8x16_narrow_i16x8(tmp0, tmp1); \
12949+ tmp = wasm_i8x16_shuffle(tmp, tmp, 0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15); \
12950+ wasm_v128_store( (void*)(ptr), tmp); \
12951+ }
12952+
12953+ #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
12954+ { \
12955+ v128_t t0 = wasm_v128_load( ptr ); \
12956+ v128_t t1 = wasm_v128_load( ptr+4 ); \
12957+ v128_t t2 = wasm_v128_load( ptr+8 ); \
12958+ v128_t t3 = wasm_v128_load( ptr+12 ); \
12959+ v128_t s0 = wasm_i32x4_shuffle(t0, t1, 0, 4, 2, 6); \
12960+ v128_t s1 = wasm_i32x4_shuffle(t0, t1, 1, 5, 3, 7); \
12961+ v128_t s2 = wasm_i32x4_shuffle(t2, t3, 0, 4, 2, 6); \
12962+ v128_t s3 = wasm_i32x4_shuffle(t2, t3, 1, 5, 3, 7); \
12963+ o0 = wasm_i32x4_shuffle(s0, s2, 0, 1, 4, 5); \
12964+ o1 = wasm_i32x4_shuffle(s1, s3, 0, 1, 4, 5); \
12965+ o2 = wasm_i32x4_shuffle(s0, s2, 2, 3, 6, 7); \
12966+ o3 = wasm_i32x4_shuffle(s1, s3, 2, 3, 6, 7); \
12967+ }
12968+
12969+ #define stbir__simdi_32shr( out, reg, imm ) out = wasm_u32x4_shr( reg, imm )
12970+
12971+ typedef float stbir__f32x4 __attribute__((__vector_size__(16), __aligned__(16)));
12972+ #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = (v128_t)(stbir__f32x4){ x, x, x, x }
12973+ #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { x, x, x, x }
12974+ #define STBIR__SIMDI_CONST4(var, a, b, c, d) stbir__simdi var = { a, b, c, d }
12975+ #define STBIR__CONSTF(var) (var)
12976+ #define STBIR__CONSTI(var) (var)
12977+
12978+ #define STBIR_SIMD
12979+
12980+#endif // SSE2/NEON/WASM
12981+
12982+#endif // NO SIMD
12983+
12984+// the X-wide names
12985+#if defined(STBIR_SIMD16)
12986+ #define STBIR__U16_DECODE_BLOCK 16 // words per block in the u16 decoders (one 32 byte load)
12987+ #define stbir__simdfX stbir__simdf16
12988+ #define stbir__simdiX stbir__simdi16
12989+ #define stbir__simdfX_load stbir__simdf16_load
12990+ #define stbir__simdiX_load stbir__simdi16_load
12991+ #define stbir__simdfX_mult stbir__simdf16_mult
12992+ #define stbir__simdfX_add_mem stbir__simdf16_add_mem
12993+ #define stbir__simdfX_madd_mem stbir__simdf16_madd_mem
12994+ #define stbir__simdfX_store stbir__simdf16_store
12995+ #define stbir__simdiX_store stbir__simdi16_store
12996+ #define stbir__simdf_frepX stbir__simdf16_frep16
12997+ #define stbir__simdfX_madd stbir__simdf16_madd
12998+ #define stbir__simdfX_min stbir__simdf16_min
12999+ #define stbir__simdfX_max stbir__simdf16_max
13000+ #define stbir__simdfX_aaa1 stbir__simdf16_aaa1
13001+ #define stbir__simdfX_1aaa stbir__simdf16_1aaa
13002+ #define stbir__simdfX_a1a1 stbir__simdf16_a1a1
13003+ #define stbir__simdfX_1a1a stbir__simdf16_1a1a
13004+ #define stbir__simdfX_convert_float_to_i32 stbir__simdf16_convert_float_to_i32
13005+ #define stbir__simdfX_pack_to_words stbir__simdf16_pack_to_32words
13006+ #define stbir__simdfX_zero stbir__simdf16_zero
13007+ #define STBIR_onesX STBIR_ones16
13008+ #define STBIR_max_uint8_as_floatX STBIR_max_uint8_as_float16
13009+ #define STBIR_max_uint16_as_floatX STBIR_max_uint16_as_float16
13010+ #define STBIR_simd_point5X STBIR_simd_point516
13011+ #define stbir__simdfX_float_count 16
13012+ #define stbir__simdfX_cast_to_simdf4( val ) stbir__simdf16_cast_to_simdf4( val )
13013+ #define stbir__simdfX_0123to1230 stbir__simdf16_0123to1230
13014+ #define stbir__simdfX_0123to2103 stbir__simdf16_0123to2103
13015+#elif defined(STBIR_SIMD8)
13016+ #define STBIR__U16_DECODE_BLOCK 8
13017+ #define stbir__simdfX stbir__simdf8
13018+ #define stbir__simdiX stbir__simdi8
13019+ #define stbir__simdfX_load stbir__simdf8_load
13020+ #define stbir__simdiX_load stbir__simdi8_load
13021+ #define stbir__simdfX_mult stbir__simdf8_mult
13022+ #define stbir__simdfX_add_mem stbir__simdf8_add_mem
13023+ #define stbir__simdfX_madd_mem stbir__simdf8_madd_mem
13024+ #define stbir__simdfX_store stbir__simdf8_store
13025+ #define stbir__simdiX_store stbir__simdi8_store
13026+ #define stbir__simdf_frepX stbir__simdf8_frep8
13027+ #define stbir__simdfX_madd stbir__simdf8_madd
13028+ #define stbir__simdfX_min stbir__simdf8_min
13029+ #define stbir__simdfX_max stbir__simdf8_max
13030+ #define stbir__simdfX_aaa1 stbir__simdf8_aaa1
13031+ #define stbir__simdfX_1aaa stbir__simdf8_1aaa
13032+ #define stbir__simdfX_a1a1 stbir__simdf8_a1a1
13033+ #define stbir__simdfX_1a1a stbir__simdf8_1a1a
13034+ #define stbir__simdfX_convert_float_to_i32 stbir__simdf8_convert_float_to_i32
13035+ #define stbir__simdfX_pack_to_words stbir__simdf8_pack_to_16words
13036+ #define stbir__simdfX_zero stbir__simdf8_zero
13037+ #define STBIR_onesX STBIR_ones8
13038+ #define STBIR_max_uint8_as_floatX STBIR_max_uint8_as_float8
13039+ #define STBIR_max_uint16_as_floatX STBIR_max_uint16_as_float8
13040+ #define STBIR_simd_point5X STBIR_simd_point58
13041+ #define stbir__simdfX_float_count 8
13042+ #define stbir__simdfX_cast_to_simdf4( val ) stbir__if_simdf8_cast_to_simdf4( val )
13043+ #define stbir__simdfX_0123to1230 stbir__simdf8_0123to12301230
13044+ #define stbir__simdfX_0123to2103 stbir__simdf8_0123to21032103
13045+#else
13046+ #define STBIR__U16_DECODE_BLOCK 8
13047+ #define stbir__simdfX stbir__simdf
13048+ #define stbir__simdiX stbir__simdi
13049+ #define stbir__simdfX_load stbir__simdf_load
13050+ #define stbir__simdiX_load stbir__simdi_load
13051+ #define stbir__simdfX_mult stbir__simdf_mult
13052+ #define stbir__simdfX_add_mem stbir__simdf_add_mem
13053+ #define stbir__simdfX_madd_mem stbir__simdf_madd_mem
13054+ #define stbir__simdfX_store stbir__simdf_store
13055+ #define stbir__simdiX_store stbir__simdi_store
13056+ #define stbir__simdf_frepX stbir__simdf_frep4
13057+ #define stbir__simdfX_madd stbir__simdf_madd
13058+ #define stbir__simdfX_min stbir__simdf_min
13059+ #define stbir__simdfX_max stbir__simdf_max
13060+ #define stbir__simdfX_aaa1 stbir__simdf_aaa1
13061+ #define stbir__simdfX_1aaa stbir__simdf_1aaa
13062+ #define stbir__simdfX_a1a1 stbir__simdf_a1a1
13063+ #define stbir__simdfX_1a1a stbir__simdf_1a1a
13064+ #define stbir__simdfX_convert_float_to_i32 stbir__simdf_convert_float_to_i32
13065+ #define stbir__simdfX_pack_to_words stbir__simdf_pack_to_8words
13066+ #define stbir__simdfX_zero stbir__simdf_zero
13067+ #define STBIR_onesX STBIR__CONSTF(STBIR_ones)
13068+ #define STBIR_simd_point5X STBIR__CONSTF(STBIR_simd_point5)
13069+ #define STBIR_max_uint8_as_floatX STBIR__CONSTF(STBIR_max_uint8_as_float)
13070+ #define STBIR_max_uint16_as_floatX STBIR__CONSTF(STBIR_max_uint16_as_float)
13071+ #define stbir__simdfX_float_count 4
13072+ #define stbir__simdfX_cast_to_simdf4( val ) ( val )
13073+ #define stbir__if_simdf8_cast_to_simdf4( val ) ( val )
13074+ #define stbir__simdfX_0123to1230 stbir__simdf_0123to1230
13075+ #define stbir__simdfX_0123to2103 stbir__simdf_0123to2103
13076+#endif
13077+
13078+
13079+#if defined(STBIR_NEON) && !defined(_M_ARM) && !defined(__arm__)
13080+
13081+ #if defined( _MSC_VER ) && !defined(__clang__)
13082+ typedef __int16 stbir__FP16;
13083+ #else
13084+ typedef float16_t stbir__FP16;
13085+ #endif
13086+
13087+#else // no NEON, or 32-bit ARM for MSVC
13088+
13089+ typedef union stbir__FP16
13090+ {
13091+ unsigned short u;
13092+ } stbir__FP16;
13093+
13094+#endif
13095+
13096+#if (!defined(STBIR_NEON) && !defined(STBIR_FP16C)) || (defined(STBIR_NEON) && defined(_M_ARM)) || (defined(STBIR_NEON) && defined(__arm__))
13097+
13098+ // Fabian's half float routines, see: https://gist.github.com/rygorous/2156668
13099+
13100+ static stbir__inline float stbir__half_to_float( stbir__FP16 h )
13101+ {
13102+ static const stbir__FP32 magic = { (254 - 15) << 23 };
13103+ static const stbir__FP32 was_infnan = { (127 + 16) << 23 };
13104+ stbir__FP32 o;
13105+
13106+ o.u = (h.u & 0x7fff) << 13; // exponent/mantissa bits
13107+ o.f *= magic.f; // exponent adjust
13108+ if (o.f >= was_infnan.f) // make sure Inf/NaN survive
13109+ o.u |= 255 << 23;
13110+ o.u |= (h.u & 0x8000) << 16; // sign bit
13111+ return o.f;
13112+ }
13113+
13114+ static stbir__inline stbir__FP16 stbir__float_to_half(float val)
13115+ {
13116+ stbir__FP32 f32infty = { 255 << 23 };
13117+ stbir__FP32 f16max = { (127 + 16) << 23 };
13118+ stbir__FP32 denorm_magic = { ((127 - 15) + (23 - 10) + 1) << 23 };
13119+ unsigned int sign_mask = 0x80000000u;
13120+ stbir__FP16 o = { 0 };
13121+ stbir__FP32 f;
13122+ unsigned int sign;
13123+
13124+ f.f = val;
13125+ sign = f.u & sign_mask;
13126+ f.u ^= sign;
13127+
13128+ if (f.u >= f16max.u) // result is Inf or NaN (all exponent bits set)
13129+ o.u = (f.u > f32infty.u) ? 0x7e00 : 0x7c00; // NaN->qNaN and Inf->Inf
13130+ else // (De)normalized number or zero
13131+ {
13132+ if (f.u < (113 << 23)) // resulting FP16 is subnormal or zero
13133+ {
13134+ // use a magic value to align our 10 mantissa bits at the bottom of
13135+ // the float. as long as FP addition is round-to-nearest-even this
13136+ // just works.
13137+ f.f += denorm_magic.f;
13138+ // and one integer subtract of the bias later, we have our final float!
13139+ o.u = (unsigned short) ( f.u - denorm_magic.u );
13140+ }
13141+ else
13142+ {
13143+ unsigned int mant_odd = (f.u >> 13) & 1; // resulting mantissa is odd
13144+ // update exponent, rounding bias part 1
13145+ f.u = f.u + ((15u - 127) << 23) + 0xfff;
13146+ // rounding bias part 2
13147+ f.u += mant_odd;
13148+ // take the bits!
13149+ o.u = (unsigned short) ( f.u >> 13 );
13150+ }
13151+ }
13152+
13153+ o.u |= sign >> 16;
13154+ return o;
13155+ }
13156+
13157+#endif
13158+
13159+
13160+#if defined(STBIR_FP16C)
13161+
13162+ #include <immintrin.h>
13163+
13164+ static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
13165+ {
13166+ _mm256_storeu_ps( (float*)output, _mm256_cvtph_ps( _mm_loadu_si128( (__m128i const* )input ) ) );
13167+ }
13168+
13169+ static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
13170+ {
13171+ _mm_storeu_si128( (__m128i*)output, _mm256_cvtps_ph( _mm256_loadu_ps( input ), 0 ) );
13172+ }
13173+
13174+ static stbir__inline float stbir__half_to_float( stbir__FP16 h )
13175+ {
13176+ return _mm_cvtss_f32( _mm_cvtph_ps( _mm_cvtsi32_si128( (int)h.u ) ) );
13177+ }
13178+
13179+ static stbir__inline stbir__FP16 stbir__float_to_half( float f )
13180+ {
13181+ stbir__FP16 h;
13182+ h.u = (unsigned short) _mm_cvtsi128_si32( _mm_cvtps_ph( _mm_set_ss( f ), 0 ) );
13183+ return h;
13184+ }
13185+
13186+#elif defined(STBIR_SSE2)
13187+
13188+ // Fabian's half float routines, see: https://gist.github.com/rygorous/2156668
13189+ stbir__inline static void stbir__half_to_float_SIMD(float * output, void const * input)
13190+ {
13191+ static const STBIR__SIMDI_CONST(mask_nosign, 0x7fff);
13192+ static const STBIR__SIMDI_CONST(smallest_normal, 0x0400);
13193+ static const STBIR__SIMDI_CONST(infinity, 0x7c00);
13194+ static const STBIR__SIMDI_CONST(expadjust_normal, (127 - 15) << 23);
13195+ static const STBIR__SIMDI_CONST(magic_denorm, 113 << 23);
13196+
13197+ __m128i i = _mm_loadu_si128 ( (__m128i const*)(input) );
13198+ __m128i h = _mm_unpacklo_epi16 ( i, _mm_setzero_si128() );
13199+ __m128i mnosign = STBIR__CONSTI(mask_nosign);
13200+ __m128i eadjust = STBIR__CONSTI(expadjust_normal);
13201+ __m128i smallest = STBIR__CONSTI(smallest_normal);
13202+ __m128i infty = STBIR__CONSTI(infinity);
13203+ __m128i expmant = _mm_and_si128(mnosign, h);
13204+ __m128i justsign = _mm_xor_si128(h, expmant);
13205+ __m128i b_notinfnan = _mm_cmpgt_epi32(infty, expmant);
13206+ __m128i b_isdenorm = _mm_cmpgt_epi32(smallest, expmant);
13207+ __m128i shifted = _mm_slli_epi32(expmant, 13);
13208+ __m128i adj_infnan = _mm_andnot_si128(b_notinfnan, eadjust);
13209+ __m128i adjusted = _mm_add_epi32(eadjust, shifted);
13210+ __m128i den1 = _mm_add_epi32(shifted, STBIR__CONSTI(magic_denorm));
13211+ __m128i adjusted2 = _mm_add_epi32(adjusted, adj_infnan);
13212+ __m128 den2 = _mm_sub_ps(_mm_castsi128_ps(den1), *(const __m128 *)&magic_denorm);
13213+ __m128 adjusted3 = _mm_and_ps(den2, _mm_castsi128_ps(b_isdenorm));
13214+ __m128 adjusted4 = _mm_andnot_ps(_mm_castsi128_ps(b_isdenorm), _mm_castsi128_ps(adjusted2));
13215+ __m128 adjusted5 = _mm_or_ps(adjusted3, adjusted4);
13216+ __m128i sign = _mm_slli_epi32(justsign, 16);
13217+ __m128 final = _mm_or_ps(adjusted5, _mm_castsi128_ps(sign));
13218+ stbir__simdf_store( output + 0, final );
13219+
13220+ h = _mm_unpackhi_epi16 ( i, _mm_setzero_si128() );
13221+ expmant = _mm_and_si128(mnosign, h);
13222+ justsign = _mm_xor_si128(h, expmant);
13223+ b_notinfnan = _mm_cmpgt_epi32(infty, expmant);
13224+ b_isdenorm = _mm_cmpgt_epi32(smallest, expmant);
13225+ shifted = _mm_slli_epi32(expmant, 13);
13226+ adj_infnan = _mm_andnot_si128(b_notinfnan, eadjust);
13227+ adjusted = _mm_add_epi32(eadjust, shifted);
13228+ den1 = _mm_add_epi32(shifted, STBIR__CONSTI(magic_denorm));
13229+ adjusted2 = _mm_add_epi32(adjusted, adj_infnan);
13230+ den2 = _mm_sub_ps(_mm_castsi128_ps(den1), *(const __m128 *)&magic_denorm);
13231+ adjusted3 = _mm_and_ps(den2, _mm_castsi128_ps(b_isdenorm));
13232+ adjusted4 = _mm_andnot_ps(_mm_castsi128_ps(b_isdenorm), _mm_castsi128_ps(adjusted2));
13233+ adjusted5 = _mm_or_ps(adjusted3, adjusted4);
13234+ sign = _mm_slli_epi32(justsign, 16);
13235+ final = _mm_or_ps(adjusted5, _mm_castsi128_ps(sign));
13236+ stbir__simdf_store( output + 4, final );
13237+
13238+ // ~38 SSE2 ops for 8 values
13239+ }
13240+
13241+ // Fabian's round-to-nearest-even float to half
13242+ // ~48 SSE2 ops for 8 output
13243+ stbir__inline static void stbir__float_to_half_SIMD(void * output, float const * input)
13244+ {
13245+ static const STBIR__SIMDI_CONST(mask_sign, 0x80000000u);
13246+ static const STBIR__SIMDI_CONST(c_f16max, (127 + 16) << 23); // all FP32 values >=this round to +inf
13247+ static const STBIR__SIMDI_CONST(c_nanbit, 0x200);
13248+ static const STBIR__SIMDI_CONST(c_infty_as_fp16, 0x7c00);
13249+ static const STBIR__SIMDI_CONST(c_min_normal, (127 - 14) << 23); // smallest FP32 that yields a normalized FP16
13250+ static const STBIR__SIMDI_CONST(c_subnorm_magic, ((127 - 15) + (23 - 10) + 1) << 23);
13251+ static const STBIR__SIMDI_CONST(c_normal_bias, 0xfff - ((127 - 15) << 23)); // adjust exponent and add mantissa rounding
13252+
13253+ __m128 f = _mm_loadu_ps(input);
13254+ __m128 msign = _mm_castsi128_ps(STBIR__CONSTI(mask_sign));
13255+ __m128 justsign = _mm_and_ps(msign, f);
13256+ __m128 absf = _mm_xor_ps(f, justsign);
13257+ __m128i absf_int = _mm_castps_si128(absf); // the cast is "free" (extra bypass latency, but no thruput hit)
13258+ __m128i f16max = STBIR__CONSTI(c_f16max);
13259+ __m128 b_isnan = _mm_cmpunord_ps(absf, absf); // is this a NaN?
13260+ __m128i b_isregular = _mm_cmpgt_epi32(f16max, absf_int); // (sub)normalized or special?
13261+ __m128i nanbit = _mm_and_si128(_mm_castps_si128(b_isnan), STBIR__CONSTI(c_nanbit));
13262+ __m128i inf_or_nan = _mm_or_si128(nanbit, STBIR__CONSTI(c_infty_as_fp16)); // output for specials
13263+
13264+ __m128i min_normal = STBIR__CONSTI(c_min_normal);
13265+ __m128i b_issub = _mm_cmpgt_epi32(min_normal, absf_int);
13266+
13267+ // "result is subnormal" path
13268+ __m128 subnorm1 = _mm_add_ps(absf, _mm_castsi128_ps(STBIR__CONSTI(c_subnorm_magic))); // magic value to round output mantissa
13269+ __m128i subnorm2 = _mm_sub_epi32(_mm_castps_si128(subnorm1), STBIR__CONSTI(c_subnorm_magic)); // subtract out bias
13270+
13271+ // "result is normal" path
13272+ __m128i mantoddbit = _mm_slli_epi32(absf_int, 31 - 13); // shift bit 13 (mantissa LSB) to sign
13273+ __m128i mantodd = _mm_srai_epi32(mantoddbit, 31); // -1 if FP16 mantissa odd, else 0
13274+
13275+ __m128i round1 = _mm_add_epi32(absf_int, STBIR__CONSTI(c_normal_bias));
13276+ __m128i round2 = _mm_sub_epi32(round1, mantodd); // if mantissa LSB odd, bias towards rounding up (RTNE)
13277+ __m128i normal = _mm_srli_epi32(round2, 13); // rounded result
13278+
13279+ // combine the two non-specials
13280+ __m128i nonspecial = _mm_or_si128(_mm_and_si128(subnorm2, b_issub), _mm_andnot_si128(b_issub, normal));
13281+
13282+ // merge in specials as well
13283+ __m128i joined = _mm_or_si128(_mm_and_si128(nonspecial, b_isregular), _mm_andnot_si128(b_isregular, inf_or_nan));
13284+
13285+ __m128i sign_shift = _mm_srai_epi32(_mm_castps_si128(justsign), 16);
13286+ __m128i final2, final= _mm_or_si128(joined, sign_shift);
13287+
13288+ f = _mm_loadu_ps(input+4);
13289+ justsign = _mm_and_ps(msign, f);
13290+ absf = _mm_xor_ps(f, justsign);
13291+ absf_int = _mm_castps_si128(absf); // the cast is "free" (extra bypass latency, but no thruput hit)
13292+ b_isnan = _mm_cmpunord_ps(absf, absf); // is this a NaN?
13293+ b_isregular = _mm_cmpgt_epi32(f16max, absf_int); // (sub)normalized or special?
13294+ nanbit = _mm_and_si128(_mm_castps_si128(b_isnan), STBIR__CONSTI(c_nanbit));
13295+ inf_or_nan = _mm_or_si128(nanbit, STBIR__CONSTI(c_infty_as_fp16)); // output for specials
13296+
13297+ b_issub = _mm_cmpgt_epi32(min_normal, absf_int);
13298+
13299+ // "result is subnormal" path
13300+ subnorm1 = _mm_add_ps(absf, _mm_castsi128_ps(STBIR__CONSTI(c_subnorm_magic))); // magic value to round output mantissa
13301+ subnorm2 = _mm_sub_epi32(_mm_castps_si128(subnorm1), STBIR__CONSTI(c_subnorm_magic)); // subtract out bias
13302+
13303+ // "result is normal" path
13304+ mantoddbit = _mm_slli_epi32(absf_int, 31 - 13); // shift bit 13 (mantissa LSB) to sign
13305+ mantodd = _mm_srai_epi32(mantoddbit, 31); // -1 if FP16 mantissa odd, else 0
13306+
13307+ round1 = _mm_add_epi32(absf_int, STBIR__CONSTI(c_normal_bias));
13308+ round2 = _mm_sub_epi32(round1, mantodd); // if mantissa LSB odd, bias towards rounding up (RTNE)
13309+ normal = _mm_srli_epi32(round2, 13); // rounded result
13310+
13311+ // combine the two non-specials
13312+ nonspecial = _mm_or_si128(_mm_and_si128(subnorm2, b_issub), _mm_andnot_si128(b_issub, normal));
13313+
13314+ // merge in specials as well
13315+ joined = _mm_or_si128(_mm_and_si128(nonspecial, b_isregular), _mm_andnot_si128(b_isregular, inf_or_nan));
13316+
13317+ sign_shift = _mm_srai_epi32(_mm_castps_si128(justsign), 16);
13318+ final2 = _mm_or_si128(joined, sign_shift);
13319+ final = _mm_packs_epi32(final, final2);
13320+ stbir__simdi_store( output,final );
13321+ }
13322+
13323+#elif defined(STBIR_NEON) && defined(_MSC_VER) && defined(STBIR__ARM64) && !defined(__clang__) // 64-bit ARM on MSVC (not clang)
13324+
13325+ static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
13326+ {
13327+ float16x4_t in0 = vld1_f16(input + 0);
13328+ float16x4_t in1 = vld1_f16(input + 4);
13329+ vst1q_f32(output + 0, vcvt_f32_f16(in0));
13330+ vst1q_f32(output + 4, vcvt_f32_f16(in1));
13331+ }
13332+
13333+ static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
13334+ {
13335+ float16x4_t out0 = vcvt_f16_f32(vld1q_f32(input + 0));
13336+ float16x4_t out1 = vcvt_f16_f32(vld1q_f32(input + 4));
13337+ vst1_f16(output+0, out0);
13338+ vst1_f16(output+4, out1);
13339+ }
13340+
13341+ static stbir__inline float stbir__half_to_float( stbir__FP16 h )
13342+ {
13343+ return vgetq_lane_f32(vcvt_f32_f16(vld1_dup_f16(&h)), 0);
13344+ }
13345+
13346+ static stbir__inline stbir__FP16 stbir__float_to_half( float f )
13347+ {
13348+ return vget_lane_f16(vcvt_f16_f32(vdupq_n_f32(f)), 0).n16_u16[0];
13349+ }
13350+
13351+#elif defined(STBIR_NEON) && defined( STBIR__ARM64 ) // 64-bit ARM
13352+
13353+ static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
13354+ {
13355+ float16x8_t in = vld1q_f16(input);
13356+ vst1q_f32(output + 0, vcvt_f32_f16(vget_low_f16(in)));
13357+ vst1q_f32(output + 4, vcvt_f32_f16(vget_high_f16(in)));
13358+ }
13359+
13360+ static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
13361+ {
13362+ float16x4_t out0 = vcvt_f16_f32(vld1q_f32(input + 0));
13363+ float16x4_t out1 = vcvt_f16_f32(vld1q_f32(input + 4));
13364+ vst1q_f16(output, vcombine_f16(out0, out1));
13365+ }
13366+
13367+ static stbir__inline float stbir__half_to_float( stbir__FP16 h )
13368+ {
13369+ return vgetq_lane_f32(vcvt_f32_f16(vdup_n_f16(h)), 0);
13370+ }
13371+
13372+ static stbir__inline stbir__FP16 stbir__float_to_half( float f )
13373+ {
13374+ return vget_lane_f16(vcvt_f16_f32(vdupq_n_f32(f)), 0);
13375+ }
13376+
13377+#elif defined(STBIR_WASM) || (defined(STBIR_NEON) && (defined(_MSC_VER) || defined(_M_ARM) || defined(__arm__))) // WASM or 32-bit ARM on MSVC/clang
13378+
13379+ static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
13380+ {
13381+ int i;
13382+ for (i=0; i<8; i++)
13383+ {
13384+ output[i] = stbir__half_to_float(input[i]);
13385+ }
13386+ }
13387+ static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
13388+ {
13389+ int i;
13390+ for (i=0; i<8; i++)
13391+ {
13392+ output[i] = stbir__float_to_half(input[i]);
13393+ }
13394+ }
13395+
13396+#endif
13397+
13398+
13399+#ifdef STBIR_SIMD
13400+
13401+#define stbir__simdf_0123to3333( out, reg ) (out) = stbir__simdf_swiz( reg, 3,3,3,3 )
13402+#define stbir__simdf_0123to2222( out, reg ) (out) = stbir__simdf_swiz( reg, 2,2,2,2 )
13403+#define stbir__simdf_0123to1111( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,1,1 )
13404+#define stbir__simdf_0123to0000( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,0,0 )
13405+#define stbir__simdf_0123to0001( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,0,1 )
13406+#define stbir__simdf_0123to1122( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,2,2 )
13407+#define stbir__simdf_0123to2333( out, reg ) (out) = stbir__simdf_swiz( reg, 2,3,3,3 )
13408+#define stbir__simdf_0123to1230( out, reg ) (out) = stbir__simdf_swiz( reg, 1,2,3,0 )
13409+#define stbir__simdf_0123to2103( out, reg ) (out) = stbir__simdf_swiz( reg, 2,1,0,3 )
13410+#define stbir__simdf_0123to3210( out, reg ) (out) = stbir__simdf_swiz( reg, 3,2,1,0 )
13411+#define stbir__simdf_0123to2301( out, reg ) (out) = stbir__simdf_swiz( reg, 2,3,0,1 )
13412+#define stbir__simdf_0123to3012( out, reg ) (out) = stbir__simdf_swiz( reg, 3,0,1,2 )
13413+#define stbir__simdf_0123to0011( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,1,1 )
13414+#define stbir__simdf_0123to2233( out, reg ) (out) = stbir__simdf_swiz( reg, 2,2,3,3 )
13415+#define stbir__simdf_0123to1133( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,3,3 )
13416+#define stbir__simdf_0123to0022( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,2,2 )
13417+#define stbir__simdf_0123to1032( out, reg ) (out) = stbir__simdf_swiz( reg, 1,0,3,2 )
13418+
13419+typedef union stbir__simdi_u32
13420+{
13421+ stbir_uint32 m128i_u32[4];
13422+ int m128i_i32[4];
13423+ stbir__simdi m128i_i128;
13424+} stbir__simdi_u32;
13425+
13426+static const int STBIR_mask[9] = { 0,0,0,-1,-1,-1,0,0,0 };
13427+
13428+static const STBIR__SIMDF_CONST(STBIR_max_uint8_as_float, stbir__max_uint8_as_float);
13429+static const STBIR__SIMDF_CONST(STBIR_max_uint16_as_float, stbir__max_uint16_as_float);
13430+static const STBIR__SIMDF_CONST(STBIR_max_uint8_as_float_inverted, stbir__max_uint8_as_float_inverted);
13431+static const STBIR__SIMDF_CONST(STBIR_max_uint16_as_float_inverted, stbir__max_uint16_as_float_inverted);
13432+
13433+static const STBIR__SIMDF_CONST(STBIR_simd_point5, 0.5f);
13434+static const STBIR__SIMDF_CONST(STBIR_ones, 1.0f);
13435+static const STBIR__SIMDI_CONST(STBIR_almost_zero, (127 - 13) << 23);
13436+static const STBIR__SIMDI_CONST(STBIR_almost_one, 0x3f7fffff);
13437+static const STBIR__SIMDI_CONST(STBIR_mantissa_mask, 0xff);
13438+static const STBIR__SIMDI_CONST(STBIR_topscale, 0x02000000);
13439+
13440+// Basically, in simd mode, we unroll the proper amount, and we don't want
13441+// the non-simd remnant loops to be unroll because they only run a few times
13442+// Adding this switch saves about 5K on clang which is Captain Unroll the 3rd.
13443+#define STBIR_SIMD_STREAMOUT_PTR( star ) STBIR_STREAMOUT_PTR( star )
13444+#define STBIR_SIMD_NO_UNROLL(ptr) STBIR_NO_UNROLL(ptr)
13445+#define STBIR_SIMD_NO_UNROLL_LOOP_START STBIR_NO_UNROLL_LOOP_START
13446+#define STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR STBIR_NO_UNROLL_LOOP_START_INF_FOR
13447+
13448+#ifdef STBIR_MEMCPY
13449+#undef STBIR_MEMCPY
13450+#endif
13451+#define STBIR_MEMCPY stbir_simd_memcpy
13452+
13453+// override normal use of memcpy with much simpler copy (faster and smaller with our sized copies)
13454+static void stbir_simd_memcpy( void * dest, void const * src, size_t bytes )
13455+{
13456+ char STBIR_SIMD_STREAMOUT_PTR (*) d = (char*) dest;
13457+ char STBIR_SIMD_STREAMOUT_PTR( * ) d_end = ((char*) dest) + bytes;
13458+ char const * s = (char const*) src;
13459+
13460+ // check overlaps
13461+ STBIR_ASSERT( ( ( d >= ( (char*)src) + bytes ) ) || ( ( d + bytes ) <= (char*)src ) );
13462+
13463+ if ( bytes < (16*stbir__simdfX_float_count) )
13464+ {
13465+ if ( bytes < 16 )
13466+ {
13467+ if ( bytes )
13468+ {
13469+ STBIR_SIMD_NO_UNROLL_LOOP_START
13470+ do
13471+ {
13472+ STBIR_SIMD_NO_UNROLL(d);
13473+ d[ 0 ] = s[ 0 ];
13474+ ++d;
13475+ ++s;
13476+ } while ( d < d_end );
13477+ }
13478+ }
13479+ else
13480+ {
13481+ stbir__simdf x;
13482+ // do one unaligned to get us aligned for the stream out below
13483+ stbir__simdf_load( x, s );
13484+ stbir__simdf_store( d, x );
13485+ s += ( ( ( (size_t)d ) + 16 ) & ~15 ) - (size_t)d;
13486+ d = (char*)( ( ( (size_t)d ) + 16 ) & ~15 );
13487+
13488+ STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
13489+ for(;;)
13490+ {
13491+ STBIR_SIMD_NO_UNROLL(d);
13492+
13493+ if ( d > ( d_end - 16 ) )
13494+ {
13495+ if ( d == d_end )
13496+ return;
13497+ s -= d - ( d_end - 16 );
13498+ d = d_end - 16;
13499+ }
13500+
13501+ stbir__simdf_load( x, s );
13502+ stbir__simdf_store( d, x );
13503+ d += 16;
13504+ s += 16;
13505+ }
13506+ }
13507+ }
13508+ else
13509+ {
13510+ stbir__simdfX x0,x1,x2,x3;
13511+
13512+ // do one unaligned to get us aligned for the stream out below
13513+ stbir__simdfX_load( x0, s + 0*stbir__simdfX_float_count );
13514+ stbir__simdfX_load( x1, s + 4*stbir__simdfX_float_count );
13515+ stbir__simdfX_load( x2, s + 8*stbir__simdfX_float_count );
13516+ stbir__simdfX_load( x3, s + 12*stbir__simdfX_float_count );
13517+ stbir__simdfX_store( d + 0*stbir__simdfX_float_count, x0 );
13518+ stbir__simdfX_store( d + 4*stbir__simdfX_float_count, x1 );
13519+ stbir__simdfX_store( d + 8*stbir__simdfX_float_count, x2 );
13520+ stbir__simdfX_store( d + 12*stbir__simdfX_float_count, x3 );
13521+ s += ( ( ( (size_t)d ) + (16*stbir__simdfX_float_count) ) & ~((16*stbir__simdfX_float_count)-1) ) - (size_t)d;
13522+ d = (char*)( ( ( (size_t)d ) + (16*stbir__simdfX_float_count) ) & ~((16*stbir__simdfX_float_count)-1) );
13523+
13524+ STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
13525+ for(;;)
13526+ {
13527+ STBIR_SIMD_NO_UNROLL(d);
13528+
13529+ if ( d > ( d_end - (16*stbir__simdfX_float_count) ) )
13530+ {
13531+ if ( d == d_end )
13532+ return;
13533+ s -= d - ( d_end - (16*stbir__simdfX_float_count) );
13534+ d = d_end - (16*stbir__simdfX_float_count);
13535+ }
13536+
13537+ stbir__simdfX_load( x0, s + 0*stbir__simdfX_float_count );
13538+ stbir__simdfX_load( x1, s + 4*stbir__simdfX_float_count );
13539+ stbir__simdfX_load( x2, s + 8*stbir__simdfX_float_count );
13540+ stbir__simdfX_load( x3, s + 12*stbir__simdfX_float_count );
13541+ stbir__simdfX_store( d + 0*stbir__simdfX_float_count, x0 );
13542+ stbir__simdfX_store( d + 4*stbir__simdfX_float_count, x1 );
13543+ stbir__simdfX_store( d + 8*stbir__simdfX_float_count, x2 );
13544+ stbir__simdfX_store( d + 12*stbir__simdfX_float_count, x3 );
13545+ d += (16*stbir__simdfX_float_count);
13546+ s += (16*stbir__simdfX_float_count);
13547+ }
13548+ }
13549+}
13550+
13551+// float copy that is specifically intentionally overlapping (src is below dest by less than the count, so a forward
13552+// copy replicates the first floats through the rest.
13553+static void stbir_overlapping_memcpy( float * dest, float const * src, size_t count )
13554+{
13555+ float STBIR_SIMD_STREAMOUT_PTR (*) sd = (float*) src;
13556+ float STBIR_SIMD_STREAMOUT_PTR( * ) s_end = ((float*) src) + count;
13557+ ptrdiff_t ofs_to_dest = dest - src;
13558+
13559+ if ( ofs_to_dest >= 4 ) // is the overlap more than 4 floats away?
13560+ {
13561+ float STBIR_SIMD_STREAMOUT_PTR( * ) s_end4 = ((float*) src) + (count&~(size_t)3);
13562+ STBIR_SIMD_NO_UNROLL_LOOP_START
13563+ do
13564+ {
13565+ stbir__simdf x;
13566+ STBIR_SIMD_NO_UNROLL(sd);
13567+ stbir__simdf_load( x, sd );
13568+ stbir__simdf_store( ( sd + ofs_to_dest ), x );
13569+ sd += 4;
13570+ } while ( sd < s_end4 );
13571+
13572+ if ( sd == s_end )
13573+ return;
13574+ }
13575+
13576+ do
13577+ {
13578+ STBIR_SIMD_NO_UNROLL(sd);
13579+ sd[ ofs_to_dest ] = sd[ 0 ];
13580+ ++sd;
13581+ } while ( sd < s_end );
13582+}
13583+
13584+#else // no SSE2
13585+
13586+// when in scalar mode, we let unrolling happen, so this macro just does the __restrict
13587+#define STBIR_SIMD_STREAMOUT_PTR( star ) STBIR_STREAMOUT_PTR( star )
13588+#define STBIR_SIMD_NO_UNROLL(ptr)
13589+#define STBIR_SIMD_NO_UNROLL_LOOP_START
13590+#define STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
13591+
13592+#endif // SSE2
13593+
13594+
13595+#ifdef STBIR_PROFILE
13596+
13597+#ifndef STBIR_PROFILE_FUNC
13598+
13599+#if ( defined(_x86_64) || defined( __x86_64__ ) || defined( _M_X64 ) || defined(__x86_64) || defined(__SSE2__) || defined(STBIR_SSE) || defined( _M_IX86_FP ) || defined(__i386) || defined( __i386__ ) || defined( _M_IX86 ) || defined( _X86_ ) ) && !defined(_M_ARM64EC)
13600+
13601+#ifdef _MSC_VER
13602+
13603+ STBIRDEF stbir_uint64 __rdtsc();
13604+ #define STBIR_PROFILE_FUNC() __rdtsc()
13605+
13606+#else // non msvc
13607+
13608+ static stbir__inline stbir_uint64 STBIR_PROFILE_FUNC()
13609+ {
13610+ stbir_uint32 lo, hi;
13611+ asm volatile ("rdtsc" : "=a" (lo), "=d" (hi) );
13612+ return ( ( (stbir_uint64) hi ) << 32 ) | ( (stbir_uint64) lo );
13613+ }
13614+
13615+#endif // msvc
13616+
13617+#elif defined( STBIR__ARM64 ) || defined(__ARM_NEON__)
13618+
13619+#if defined( _MSC_VER ) && !defined(__clang__)
13620+
13621+ #include <intrin.h> // _ReadStatusReg, ARM64_SYSREG
13622+ #ifndef ARM64_CNTVCT // winnt.h defines it; without windows.h build the same encoding (CNTVCT_EL0, the generic timer counter)
13623+ #define ARM64_CNTVCT ARM64_SYSREG(3,3,14,0,2)
13624+ #endif
13625+ #define STBIR_PROFILE_FUNC() _ReadStatusReg(ARM64_CNTVCT)
13626+
13627+#else
13628+
13629+ static stbir__inline stbir_uint64 STBIR_PROFILE_FUNC()
13630+ {
13631+ stbir_uint64 tsc;
13632+ asm volatile("mrs %0, cntvct_el0" : "=r" (tsc));
13633+ return tsc;
13634+ }
13635+
13636+#endif
13637+
13638+#else // x64, arm
13639+
13640+#error Unknown platform for profiling.
13641+
13642+#endif // x64, arm
13643+
13644+#endif // STBIR_PROFILE_FUNC
13645+
13646+#define STBIR_ONLY_PROFILE_GET_SPLIT_INFO ,stbir__per_split_info * split_info
13647+#define STBIR_ONLY_PROFILE_SET_SPLIT_INFO ,split_info
13648+
13649+#define STBIR_ONLY_PROFILE_BUILD_GET_INFO ,stbir__info * profile_info
13650+#define STBIR_ONLY_PROFILE_BUILD_SET_INFO ,profile_info
13651+
13652+// super light-weight micro profiler
13653+#define STBIR_PROFILE_START_ll( info, wh ) { stbir_uint64 wh##thiszonetime = STBIR_PROFILE_FUNC(); stbir_uint64 * wh##save_parent_excluded_ptr = info->current_zone_excluded_ptr; stbir_uint64 wh##current_zone_excluded = 0; info->current_zone_excluded_ptr = &wh##current_zone_excluded;
13654+#define STBIR_PROFILE_END_ll( info, wh ) wh##thiszonetime = STBIR_PROFILE_FUNC() - wh##thiszonetime; info->profile.named.wh += wh##thiszonetime - wh##current_zone_excluded; *wh##save_parent_excluded_ptr += wh##thiszonetime; info->current_zone_excluded_ptr = wh##save_parent_excluded_ptr; }
13655+#define STBIR_PROFILE_FIRST_START_ll( info, wh ) { int zone; info->current_zone_excluded_ptr = &info->profile.named.total; for(zone=0;zone<(int)STBIR__ARRAY_SIZE(info->profile.array);zone++) info->profile.array[zone]=0; } STBIR_PROFILE_START_ll( info, wh );
13656+#define STBIR_PROFILE_CLEAR_EXTRAS_ll( info, num ) { int extra; for(extra=1;extra<(num);extra++) { int zone; for(zone=0;zone<(int)STBIR__ARRAY_SIZE((info)->profile.array);zone++) (info)[extra].profile.array[zone]=0; } }
13657+
13658+// for thread data
13659+#define STBIR_PROFILE_START( wh ) STBIR_PROFILE_START_ll( split_info, wh )
13660+#define STBIR_PROFILE_END( wh ) STBIR_PROFILE_END_ll( split_info, wh )
13661+#define STBIR_PROFILE_FIRST_START( wh ) STBIR_PROFILE_FIRST_START_ll( split_info, wh )
13662+#define STBIR_PROFILE_CLEAR_EXTRAS() STBIR_PROFILE_CLEAR_EXTRAS_ll( split_info, split_count )
13663+#define STBIR_PROFILE_NEST_IN_FIRST( extra ) { if ( ( extra ) != split_info ) ( extra )->current_zone_excluded_ptr = split_info->current_zone_excluded_ptr; } // an extra split's zones count against the first split's open zone
13664+
13665+// for build data
13666+#define STBIR_PROFILE_BUILD_START( wh ) STBIR_PROFILE_START_ll( profile_info, wh )
13667+#define STBIR_PROFILE_BUILD_END( wh ) STBIR_PROFILE_END_ll( profile_info, wh )
13668+#define STBIR_PROFILE_BUILD_FIRST_START( wh ) STBIR_PROFILE_FIRST_START_ll( profile_info, wh )
13669+#define STBIR_PROFILE_BUILD_CLEAR( info ) { int zone; for(zone=0;zone<(int)STBIR__ARRAY_SIZE(info->profile.array);zone++) info->profile.array[zone]=0; }
13670+
13671+#else // no profile
13672+
13673+#define STBIR_ONLY_PROFILE_GET_SPLIT_INFO
13674+#define STBIR_ONLY_PROFILE_SET_SPLIT_INFO
13675+
13676+#define STBIR_ONLY_PROFILE_BUILD_GET_INFO
13677+#define STBIR_ONLY_PROFILE_BUILD_SET_INFO
13678+
13679+#define STBIR_PROFILE_START( wh )
13680+#define STBIR_PROFILE_END( wh )
13681+#define STBIR_PROFILE_FIRST_START( wh )
13682+#define STBIR_PROFILE_CLEAR_EXTRAS( )
13683+#define STBIR_PROFILE_NEST_IN_FIRST( extra )
13684+
13685+#define STBIR_PROFILE_BUILD_START( wh )
13686+#define STBIR_PROFILE_BUILD_END( wh )
13687+#define STBIR_PROFILE_BUILD_FIRST_START( wh )
13688+#define STBIR_PROFILE_BUILD_CLEAR( info )
13689+
13690+#endif // stbir_profile
13691+
13692+#ifndef STBIR_MEMCPY
13693+// For memcpy
13694+#include <string.h>
13695+#define STBIR_MEMCPY( dest, src, len ) memcpy( dest, src, len )
13696+#endif
13697+
13698+#ifndef STBIR_SIMD
13699+
13700+// float copy that is specifically intentionally overlapping (src is below dest by less than the count, so a forward
13701+// copy replicates the first floats through the rest.
13702+static void stbir_overlapping_memcpy( float * dest, float const * src, size_t count )
13703+{
13704+ float STBIR_SIMD_STREAMOUT_PTR (*) sd = (float*) src;
13705+ float STBIR_SIMD_STREAMOUT_PTR( * ) s_end = ((float*) src) + count;
13706+ ptrdiff_t ofs_to_dest = dest - src;
13707+
13708+ if ( ofs_to_dest >= 2 ) // is the overlap more than 2 floats away
13709+ {
13710+ float STBIR_SIMD_STREAMOUT_PTR( * ) s_end2 = ((float*) src) + (count&~(size_t)1);
13711+
13712+ STBIR_NO_UNROLL_LOOP_START
13713+ do
13714+ {
13715+ float a,b;
13716+ STBIR_NO_UNROLL(sd);
13717+ a = sd[0];
13718+ b = sd[1];
13719+ sd[ ofs_to_dest ] = a;
13720+ sd[ ofs_to_dest + 1 ] = b;
13721+ sd += 2;
13722+ } while ( sd < s_end2 );
13723+
13724+ if ( sd == s_end )
13725+ return;
13726+ }
13727+
13728+ STBIR_NO_UNROLL_LOOP_START
13729+ do
13730+ {
13731+ STBIR_NO_UNROLL(sd);
13732+ sd[ ofs_to_dest ] = sd[ 0 ];
13733+ ++sd;
13734+ } while ( sd < s_end );
13735+}
13736+
13737+#endif
13738+
13739+// the built-in kernels (all in stbir_real; the generator rounds the weight to float once, on the way into the
13740+// coefficient table). The box filter as the overlap of a pixel with the output pixel's footprint: 1 within
13741+// 0.5 - scale / 2 of the center, 0 beyond 0.5 + scale / 2, a ramp between.
13742+static stbir_real stbir__filter_trapezoid(stbir_real x, stbir_real scale, void * user_data)
13743+{
13744+ stbir_real halfscale = scale * (stbir_real)0.5;
13745+ stbir_real t = (stbir_real)0.5 + halfscale;
13746+ stbir_real r = (stbir_real)0.5 - halfscale;
13747+ STBIR_ASSERT(scale <= (stbir_real)1.0);
13748+ STBIR__UNUSED(user_data);
13749+
13750+ if ( x < (stbir_real) 0.0 ) x = -x;
13751+
13752+ if ( x >= t )
13753+ return (stbir_real)0.0;
13754+ if ( x <= r )
13755+ return (stbir_real)1.0;
13756+ return ( t - x ) / scale;
13757+}
13758+
13759+// the box's extent depends on the scale, so it is given as a radius (in input pixels): one multiply, so an integer
13760+// ratio's tap count is exact
13761+static stbir_real stbir__radius_trapezoid(stbir_real scale, stbir_real inv_scale, int is_upsample, void * user_data)
13762+{
13763+ STBIR__UNUSED(scale);
13764+ STBIR__UNUSED(is_upsample);
13765+ STBIR__UNUSED(user_data);
13766+ return (stbir_real)0.5 * inv_scale + (stbir_real)0.5; // ( 0.5 + scale / 2 ) output pixels downsampling, 0.5 + inv_scale / 2 input pixels upsampling: the same number
13767+}
13768+
13769+static stbir_real stbir__filter_triangle(stbir_real x, stbir_real s, void * user_data)
13770+{
13771+ STBIR__UNUSED(s);
13772+ STBIR__UNUSED(user_data);
13773+
13774+ if ( x < (stbir_real) 0.0 ) x = -x;
13775+
13776+ if (x <= (stbir_real) 1.0)
13777+ return (stbir_real) 1.0 - x;
13778+ else
13779+ return (stbir_real) 0.0;
13780+}
13781+
13782+static stbir_real stbir__filter_point(stbir_real x, stbir_real s, void * user_data)
13783+{
13784+ STBIR__UNUSED(x);
13785+ STBIR__UNUSED(s);
13786+ STBIR__UNUSED(user_data);
13787+
13788+ return (stbir_real) 1.0;
13789+}
13790+
13791+// the Mitchell-Netrevalli cubics with C = (1-B)/2: B=1 is the cubic b-spline, 1/3 Mitchell, 0 Catmull-Rom
13792+static stbir__inline stbir_real stbir__mitchell_netrevalli(stbir_real x, stbir_real B)
13793+{
13794+ if ( x < (stbir_real) 0.0 ) x = -x;
13795+
13796+ if (x < (stbir_real) 1.0)
13797+ return (((stbir_real) 6.0 - (stbir_real) 2.0*B) + x*x*(((stbir_real) 9.0 - (stbir_real) 6.0*B)*x - ((stbir_real) 15.0 - (stbir_real) 9.0*B)))/(stbir_real) 6.0;
13798+ else if (x < (stbir_real) 2.0)
13799+ return (((stbir_real) 12.0 - (stbir_real) 4.0*B) + x*(((stbir_real) 12.0*B - (stbir_real) 24.0) + x*(((stbir_real) 15.0 - (stbir_real) 9.0*B) + ((stbir_real) 2.0*B - (stbir_real) 3.0)*x)))/(stbir_real) 6.0;
13800+
13801+ return (stbir_real) 0.0;
13802+}
13803+
13804+static stbir_real stbir__filter_cubic(stbir_real x, stbir_real s, void * user_data)
13805+{
13806+ STBIR__UNUSED(s);
13807+ STBIR__UNUSED(user_data);
13808+
13809+ return stbir__mitchell_netrevalli( x, (stbir_real) 1.0 );
13810+}
13811+
13812+// Catmull-Rom is the family at B=0, written out (it's the upsampling default)
13813+static stbir_real stbir__filter_catmullrom(stbir_real x, stbir_real s, void * user_data)
13814+{
13815+ STBIR__UNUSED(s);
13816+ STBIR__UNUSED(user_data);
13817+
13818+ if ( x < (stbir_real) 0.0 ) x = -x;
13819+
13820+ if (x < (stbir_real) 1.0)
13821+ return (stbir_real) 1.0 - x*x*((stbir_real) 2.5 - (stbir_real) 1.5*x);
13822+ else if (x < (stbir_real) 2.0)
13823+ return (stbir_real) 2.0 - x*((stbir_real) 4.0 + x*((stbir_real) 0.5*x - (stbir_real) 2.5));
13824+
13825+ return (stbir_real) 0.0;
13826+}
13827+
13828+static stbir_real stbir__filter_mitchell(stbir_real x, stbir_real s, void * user_data)
13829+{
13830+ STBIR__UNUSED(s);
13831+ STBIR__UNUSED(user_data);
13832+
13833+ return stbir__mitchell_netrevalli( x, (stbir_real) 1.0 / (stbir_real) 3.0 );
13834+}
13835+
13836+static stbir_real stbir__filter_mitchell_quarter(stbir_real x, stbir_real s, void * user_data)
13837+{
13838+ STBIR__UNUSED(s);
13839+ STBIR__UNUSED(user_data);
13840+
13841+ return stbir__mitchell_netrevalli( x, (stbir_real) 0.25 );
13842+}
13843+
13844+static stbir_real stbir__filter_mitchell_sixth(stbir_real x, stbir_real s, void * user_data)
13845+{
13846+ STBIR__UNUSED(s);
13847+ STBIR__UNUSED(user_data);
13848+
13849+ return stbir__mitchell_netrevalli( x, (stbir_real) 1.0 / (stbir_real) 6.0 );
13850+}
13851+
13852+static stbir_real stbir__filter_mitchell_neg_sixth(stbir_real x, stbir_real s, void * user_data)
13853+{
13854+ STBIR__UNUSED(s);
13855+ STBIR__UNUSED(user_data);
13856+
13857+ return stbir__mitchell_netrevalli( x, (stbir_real) -1.0 / (stbir_real) 6.0 );
13858+}
13859+
13860+static stbir_real stbir__filter_mitchell_neg_third(stbir_real x, stbir_real s, void * user_data)
13861+{
13862+ STBIR__UNUSED(s);
13863+ STBIR__UNUSED(user_data);
13864+
13865+ return stbir__mitchell_netrevalli( x, (stbir_real) -1.0 / (stbir_real) 3.0 );
13866+}
13867+
13868+// sin(pi x) / (pi x), done here so the coefficients match on every platform (no libm). sin(pi x) flips sign at every
13869+// whole x, so reduce to the nearest whole and run the series on [-pi/2, pi/2] (good to ~1e-9).
13870+static stbir_real stbir__sinc_pi(stbir_real x)
13871+{
13872+ stbir_real const pi = (stbir_real) 3.14159265358979323846;
13873+ stbir_real u, u2, s;
13874+ int n;
13875+
13876+ if ( x < (stbir_real) 0.0 ) x = -x;
13877+
13878+ if (x < (stbir_real) 1e-4)
13879+ {
13880+ stbir_real px = pi * x;
13881+ return (stbir_real) 1.0 - px*px / (stbir_real) 6.0;
13882+ }
13883+
13884+ n = (int) ( x + (stbir_real) 0.5 );
13885+ u = pi * ( x - (stbir_real) n );
13886+ u2 = u*u;
13887+ s = (stbir_real) 1.0 - u2 / (stbir_real) 156.0; // the series from the inside out: 1 - u^2 / ( 2k ( 2k + 1 ) )
13888+ s = (stbir_real) 1.0 - u2 / (stbir_real) 110.0 * s;
13889+ s = (stbir_real) 1.0 - u2 / (stbir_real) 72.0 * s;
13890+ s = (stbir_real) 1.0 - u2 / (stbir_real) 42.0 * s;
13891+ s = (stbir_real) 1.0 - u2 / (stbir_real) 20.0 * s;
13892+ s = (stbir_real) 1.0 - u2 / (stbir_real) 6.0 * s;
13893+ s = u * s;
13894+ if ( n & 1 ) s = -s;
13895+
13896+ return s / ( pi * x );
13897+}
13898+
13899+// Lanczos: a sinc windowed by a sinc stretched to the support
13900+static stbir_real stbir__filter_lanczos(stbir_real x, stbir_real support)
13901+{
13902+ if ( x < (stbir_real) 0.0 ) x = -x;
13903+
13904+ if (x < support)
13905+ return stbir__sinc_pi( x ) * stbir__sinc_pi( x / support );
13906+
13907+ return (stbir_real) 0.0;
13908+}
13909+
13910+static stbir_real stbir__filter_lanczos4(stbir_real x, stbir_real s, void * user_data)
13911+{
13912+ STBIR__UNUSED(s);
13913+ STBIR__UNUSED(user_data);
13914+
13915+ return stbir__filter_lanczos( x, (stbir_real) 2.0 );
13916+}
13917+
13918+static stbir_real stbir__filter_lanczos5(stbir_real x, stbir_real s, void * user_data)
13919+{
13920+ STBIR__UNUSED(s);
13921+ STBIR__UNUSED(user_data);
13922+
13923+ return stbir__filter_lanczos( x, (stbir_real) 2.5 );
13924+}
13925+
13926+// Casey Muratori's Bink 2 half pixel kernel ( 1 -4 19 19 -4 1 ) / 32: a Catmull-Rom spline through knots every half
13927+// pixel (1 at 0, the weights at the half pixels, 0 at the whole ones), padded at both ends so the four points stay in the table
13928+static stbir_real stbir__filter_muratori6(stbir_real x, stbir_real s, void * user_data)
13929+{
13930+ static stbir_real const knots[9] = { (stbir_real) 19.0/(stbir_real) 32.0, (stbir_real) 1.0, (stbir_real) 19.0/(stbir_real) 32.0, (stbir_real) 0.0, (stbir_real) -4.0/(stbir_real) 32.0, (stbir_real) 0.0, (stbir_real) 1.0/(stbir_real) 32.0, (stbir_real) 0.0, (stbir_real) 0.0 };
13931+ stbir_real t, p0, p1, p2, p3;
13932+ int i;
13933+ STBIR__UNUSED(s);
13934+ STBIR__UNUSED(user_data);
13935+
13936+ if ( x < (stbir_real) 0.0 ) x = -x;
13937+
13938+ if ( x >= (stbir_real) 3.0 )
13939+ return (stbir_real) 0.0;
13940+
13941+ t = x * (stbir_real) 2.0;
13942+ i = (int) t;
13943+ t -= (stbir_real) i;
13944+
13945+ p0 = knots[i];
13946+ p1 = knots[i+1];
13947+ p2 = knots[i+2];
13948+ p3 = knots[i+3];
13949+
13950+ return (stbir_real) 0.5*((stbir_real) 2.0*p1 + t*((p2 - p0) + t*(((stbir_real) 2.0*p0 - (stbir_real) 5.0*p1 + (stbir_real) 4.0*p2 - p3) + t*(((stbir_real) 3.0*p1 - (stbir_real) 3.0*p2) + (p3 - p0)))));
13951+}
13952+
13953+// point sampling takes the one nearest input pixel in both directions.
13954+static stbir_real stbir__radius_point(stbir_real scale, stbir_real inv_scale, int is_upsample, void * user_data)
13955+{
13956+ STBIR__UNUSED(scale);
13957+ STBIR__UNUSED(inv_scale);
13958+ STBIR__UNUSED(is_upsample);
13959+ STBIR__UNUSED(user_data);
13960+ return (stbir_real)0.5;
13961+}
13962+
13963+// the other built-ins' supports: the kernel's radius in filter pixels
13964+
13965+static stbir_real stbir__support_one(stbir_real scale, void * user_data)
13966+{
13967+ STBIR__UNUSED(scale);
13968+ STBIR__UNUSED(user_data);
13969+ return (stbir_real)1.0;
13970+}
13971+
13972+static stbir_real stbir__support_two(stbir_real scale, void * user_data)
13973+{
13974+ STBIR__UNUSED(scale);
13975+ STBIR__UNUSED(user_data);
13976+ return (stbir_real)2.0;
13977+}
13978+
13979+static stbir_real stbir__support_two_and_half(stbir_real scale, void * user_data)
13980+{
13981+ STBIR__UNUSED(scale);
13982+ STBIR__UNUSED(user_data);
13983+ return (stbir_real)2.5;
13984+}
13985+
13986+static stbir_real stbir__support_three(stbir_real scale, void * user_data)
13987+{
13988+ STBIR__UNUSED(scale);
13989+ STBIR__UNUSED(user_data);
13990+ return (stbir_real)3.0;
13991+}
13992+
13993+// the built-in filters, indexed by stbir_filter (everything below STBIR_FILTER_OTHER
13994+static stbir__kernel_callback_new * stbir__builtin_kernels[STBIR_FILTER_OTHER] =
13995+{
13996+ 0, // STBIR_FILTER_DEFAULT (resolved before the lookup)
13997+ stbir__filter_trapezoid, // STBIR_FILTER_BOX
13998+ stbir__filter_triangle, // STBIR_FILTER_TRIANGLE
13999+ stbir__filter_cubic, // STBIR_FILTER_CUBICBSPLINE
14000+ stbir__filter_catmullrom, // STBIR_FILTER_CATMULLROM
14001+ stbir__filter_mitchell, // STBIR_FILTER_MITCHELL
14002+ stbir__filter_point, // STBIR_FILTER_POINT_SAMPLE
14003+ stbir__filter_mitchell_quarter, // STBIR_FILTER_MITCHELL_QUARTER
14004+ stbir__filter_mitchell_sixth, // STBIR_FILTER_MITCHELL_SIXTH
14005+ stbir__filter_mitchell_neg_sixth, // STBIR_FILTER_MITCHELL_NEG_SIXTH
14006+ stbir__filter_mitchell_neg_third, // STBIR_FILTER_MITCHELL_NEG_THIRD
14007+ stbir__filter_lanczos4, // STBIR_FILTER_LANCZOS4
14008+ stbir__filter_lanczos5, // STBIR_FILTER_LANCZOS5
14009+ stbir__filter_muratori6, // STBIR_FILTER_MURATORI6
14010+};
14011+
14012+static stbir__support_callback_new * stbir__builtin_supports[STBIR_FILTER_OTHER] =
14013+{
14014+ 0, // STBIR_FILTER_DEFAULT
14015+ 0, // STBIR_FILTER_BOX (a radius)
14016+ stbir__support_one, // STBIR_FILTER_TRIANGLE
14017+ stbir__support_two, // STBIR_FILTER_CUBICBSPLINE
14018+ stbir__support_two, // STBIR_FILTER_CATMULLROM
14019+ stbir__support_two, // STBIR_FILTER_MITCHELL
14020+ 0, // STBIR_FILTER_POINT_SAMPLE (a radius)
14021+ stbir__support_two, // STBIR_FILTER_MITCHELL_QUARTER
14022+ stbir__support_two, // STBIR_FILTER_MITCHELL_SIXTH
14023+ stbir__support_two, // STBIR_FILTER_MITCHELL_NEG_SIXTH
14024+ stbir__support_two, // STBIR_FILTER_MITCHELL_NEG_THIRD
14025+ stbir__support_two, // STBIR_FILTER_LANCZOS4
14026+ stbir__support_two_and_half, // STBIR_FILTER_LANCZOS5
14027+ stbir__support_three, // STBIR_FILTER_MURATORI6
14028+};
14029+
14030+static stbir__radius_callback_new * stbir__builtin_radii[STBIR_FILTER_OTHER] =
14031+{
14032+ 0, // STBIR_FILTER_DEFAULT
14033+ stbir__radius_trapezoid, // STBIR_FILTER_BOX
14034+ 0, 0, 0, 0, // STBIR_FILTER_TRIANGLE .. STBIR_FILTER_MITCHELL (supports)
14035+ stbir__radius_point, // STBIR_FILTER_POINT_SAMPLE
14036+ 0, 0, 0, 0, 0, 0, 0, // STBIR_FILTER_MITCHELL_QUARTER .. STBIR_FILTER_MURATORI6 (supports)
14037+};
14038+
14039+// upsample (1:1 included), and a 1:1 with the output on the input's pixel grid (a copy)
14040+static int stbir__scale_is_upsample( stbir__scale_info const * scale_info )
14041+{
14042+ return ( scale_info->scale >= (stbir_real) 1.0 );
14043+}
14044+
14045+static int stbir__scale_is_copy( stbir__scale_info const * scale_info )
14046+{
14047+ return ( scale_info->positions_exact ) && ( scale_info->scale_numerator == scale_info->scale_denominator );
14048+}
14049+
14050+// the sum under which a coefficient set is zeroed
14051+static stbir_real stbir__too_small_coefficient_sum_for( stbir__scale_info const * scale_info )
14052+{
14053+ if ( stbir__scale_is_upsample( scale_info ) )
14054+ return stbir__too_small_coefficient_sum;
14055+ return stbir__too_small_coefficient_sum * scale_info->scale;
14056+}
14057+
14058+// floor and ceil without math.h (the header doesn't use it), either sign, for pixel coordinates
14059+// and tap counts - the range is known to fit an int
14060+static int stbir__floor_to_int( stbir_real x )
14061+{
14062+ int i = (int) x;
14063+ return ( (stbir_real)i > x ) ? i - 1 : i;
14064+}
14065+
14066+static int stbir__ceil_to_int( stbir_real x )
14067+{
14068+ int i = (int) x;
14069+ return ( (stbir_real)i < x ) ? i + 1 : i;
14070+}
14071+
14072+// the filter's radius in input pixels. a radius callback gives it directly. a support callback gives the radius in
14073+// filter pixels (called with the ratio <= 1): input pixels when upsampling, output pixels when downsampling, so it
14074+// is scaled by inv_scale there. the old float form is converted the same way.
14075+static stbir_real stbir__filter_radius_in_input_pixels( stbir__radius_callback_new * radius, stbir__support_callback_new * support_new, stbir__support_callback * support_old, stbir__scale_info const * scale_info, void * user_data )
14076+{
14077+ int is_upsample = stbir__scale_is_upsample( scale_info );
14078+ stbir_real r;
14079+
14080+ if ( radius )
14081+ r = radius( scale_info->scale, scale_info->inv_scale, is_upsample, user_data );
14082+ else if ( is_upsample )
14083+ {
14084+ if ( support_new )
14085+ r = support_new( scale_info->inv_scale, user_data );
14086+ else
14087+ r = (stbir_real) support_old( (float) scale_info->inv_scale, user_data );
14088+ }
14089+ else
14090+ {
14091+ if ( support_new )
14092+ r = support_new( scale_info->scale, user_data );
14093+ else
14094+ r = (stbir_real) support_old( (float) scale_info->scale, user_data );
14095+ r *= scale_info->inv_scale;
14096+ }
14097+
14098+ // at least half an input pixel (point sampling's), so every output has a tap: the downsample generator walks the
14099+ // inputs and gives each output the ones within reach. a user filter's might be under; the built-ins never are
14100+ if ( !( r >= (stbir_real)0.5 ) )
14101+ r = (stbir_real)0.5;
14102+
14103+ return r;
14104+}
14105+
14106+static stbir_real stbir__filter_radius_in_output_pixels( stbir__radius_callback_new * radius, stbir__support_callback_new * support_new, stbir__support_callback * support_old, stbir__scale_info const * scale_info, void * user_data )
14107+{
14108+ return stbir__filter_radius_in_input_pixels( radius, support_new, support_old, scale_info, user_data ) / scale_info->inv_scale;
14109+}
14110+
14111+// This is the maximum number of input samples that can affect an output sample
14112+// with the given filter from the output pixel's perspective
14113+static int stbir__get_filter_pixel_width( stbir__radius_callback_new * radius, stbir__support_callback_new * support_new, stbir__support_callback * support_old, stbir__scale_info const * scale_info, void * user_data)
14114+{
14115+ STBIR_ASSERT( ( radius != 0 ) || ( support_new != 0 ) || ( support_old != 0 ) );
14116+ return stbir__ceil_to_int( (stbir_real)2.0 * stbir__filter_radius_in_input_pixels( radius, support_new, support_old, scale_info, user_data ) );
14117+}
14118+
14119+// this is how many coefficients per run of the filter (which is different from the filter_pixel_width depending on if we are scattering or gathering)
14120+static int stbir__get_coefficient_width(stbir__sampler * samp, int is_gather, void * user_data)
14121+{
14122+ stbir__scale_info const * scale_info = &samp->scale_info;
14123+
14124+ switch( is_gather )
14125+ {
14126+ case 1:
14127+ case 2:
14128+ // one of slack: a tap exactly on the far edge is in the range (stbir__exact_pixel_range), one more than ceil( 2 x radius )
14129+ return stbir__ceil_to_int( (stbir_real)2.0 * stbir__filter_radius_in_input_pixels( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, scale_info, user_data ) ) + 1;
14130+ case 0:
14131+ return stbir__ceil_to_int( (stbir_real)2.0 * stbir__filter_radius_in_output_pixels( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, scale_info, user_data ) );
14132+ default:
14133+ STBIR_ASSERT( (is_gather >= 0 ) && (is_gather <= 2 ) );
14134+ return 0;
14135+ }
14136+}
14137+
14138+static int stbir__get_contributors(stbir__sampler * samp, int is_gather)
14139+{
14140+ if (is_gather)
14141+ return samp->scale_info.output_sub_size;
14142+ else
14143+ return (samp->scale_info.input_full_size + samp->filter_pixel_margin * 2);
14144+}
14145+
14146+static int stbir__edge_zero_full( int n, int max )
14147+{
14148+ STBIR__UNUSED(n);
14149+ STBIR__UNUSED(max);
14150+ return 0; // NOTREACHED
14151+}
14152+
14153+static int stbir__edge_clamp_full( int n, int max )
14154+{
14155+ if (n < 0)
14156+ return 0;
14157+
14158+ if (n >= max)
14159+ return max - 1;
14160+
14161+ return n; // NOTREACHED
14162+}
14163+
14164+static int stbir__edge_reflect_full( int n, int max )
14165+{
14166+ if (n < 0)
14167+ {
14168+ if (n > -max)
14169+ return -n;
14170+ else
14171+ return max - 1;
14172+ }
14173+
14174+ if (n >= max)
14175+ {
14176+ if ( ( n - max ) >= max ) // (as n >= 2 max, without the overflow)
14177+ return 0;
14178+ else
14179+ return ( max - 1 ) - ( n - max );
14180+ }
14181+
14182+ return n; // NOTREACHED
14183+}
14184+
14185+static int stbir__edge_wrap_full( int n, int max )
14186+{
14187+ if (n >= 0)
14188+ return (n % max);
14189+ else
14190+ {
14191+ int m = (-n) % max;
14192+
14193+ if (m != 0)
14194+ m = max - m;
14195+
14196+ return (m);
14197+ }
14198+}
14199+
14200+typedef int stbir__edge_wrap_func( int n, int max );
14201+static stbir__edge_wrap_func * stbir__edge_wrap_slow[] =
14202+{
14203+ stbir__edge_clamp_full, // STBIR_EDGE_CLAMP
14204+ stbir__edge_reflect_full, // STBIR_EDGE_REFLECT
14205+ stbir__edge_wrap_full, // STBIR_EDGE_WRAP
14206+ stbir__edge_zero_full, // STBIR_EDGE_ZERO
14207+};
14208+
14209+stbir__inline static int stbir__edge_wrap(stbir_edge edge, int n, int max)
14210+{
14211+ // avoid per-pixel switch
14212+ if (n >= 0 && n < max)
14213+ return n;
14214+ return stbir__edge_wrap_slow[edge]( n, max );
14215+}
14216+
14217+// the ranges' edge handling: in pixel ranges may hang off the image with WRAP (the decoder wraps them, within a width
14218+// each side), out pixel ranges are clipped to the output
14219+static void stbir__clip_in_pixel_range( int first, int last, int input_size, stbir_edge edge, int * first_pixel, int * last_pixel )
14220+{
14221+ if ( last < first )
14222+ last = first; // point sample mode can span a value *right* at 0.5, and cause these to cross
14223+ if ( edge == STBIR_EDGE_WRAP )
14224+ {
14225+ if ( first < -input_size )
14226+ first = -input_size;
14227+ if ( ( last - input_size ) >= input_size ) // (as last >= 2 input_size, without the overflow)
14228+ last = ( input_size - 1 ) + input_size;
14229+ }
14230+ *first_pixel = first;
14231+ *last_pixel = last;
14232+}
14233+
14234+static void stbir__clip_out_pixel_range( int first, int last, int out_size, int * first_pixel, int * last_pixel )
14235+{
14236+ if ( first < 0 )
14237+ first = 0;
14238+ if ( last >= out_size )
14239+ last = out_size - 1;
14240+ *first_pixel = first;
14241+ *last_pixel = last;
14242+}
14243+
14244+// The pixels within support of the center numer / den (quotient kept apart, only the remainder meets the real math).
14245+// On the far edge is in, on the near edge out; stbir__tie_epsilon lifts the tie above rounding noise (point sampling
14246+// hits both edges at every even ratio). The output range and the real chain ranges below use the same rule.
14247+#define stbir__tie_epsilon ( ( sizeof( stbir_real ) == 4 ) ? (stbir_real)( 1.0 / ( 1 << 18 ) ) : (stbir_real)( 1.0 / ( (stbir_int64)1 << 36 ) ) )
14248+
14249+// The input pixels within out_filter_radius (output pixels) of an output pixel's center.
14250+static void stbir__calculate_in_pixel_range( int * first_pixel, int * last_pixel, stbir_real out_pixel_center, stbir_real out_filter_radius, stbir_real inv_scale, stbir_real out_shift, int input_size, stbir_edge edge )
14251+{
14252+ stbir_real in_pixel_influence_lowerbound = ( out_pixel_center - out_filter_radius + out_shift ) * inv_scale;
14253+ stbir_real in_pixel_influence_upperbound = ( out_pixel_center + out_filter_radius + out_shift ) * inv_scale;
14254+ stbir__clip_in_pixel_range( stbir__floor_to_int( in_pixel_influence_lowerbound + (stbir_real)0.5 + stbir__tie_epsilon ), stbir__floor_to_int( in_pixel_influence_upperbound - (stbir_real)0.5 + stbir__tie_epsilon ), input_size, edge, first_pixel, last_pixel );
14255+}
14256+
14257+// Output pixel o's center against input pixel i's (in input pixels for upsamples, output pixels for downsamples). With an
14258+// exact scale N/D and integer shift it is ( ( 2o + 1 + 2 shift ) D - ( 2i + 1 ) N ) / 2N (or 2D): integers until one
14259+// multiply by the precomputed 1 / 2N (or 1 / 2D), exact to float at any coordinate.
14260+static stbir_real stbir__tap_offset( stbir__scale_info const * scale_info, int o, int i, int in_input_pixels )
14261+{
14262+ if ( scale_info->positions_exact )
14263+ {
14264+ // (the difference is at most the filter's width times 2N or 2D: exact in a double, a float ulp in a float)
14265+ stbir_int64 numer = ( 2 * (stbir_int64)o + 1 + 2 * (stbir_int64)scale_info->shift_pixels ) * (stbir_int64)scale_info->scale_denominator
14266+ - ( 2 * (stbir_int64)i + 1 ) * (stbir_int64)scale_info->scale_numerator;
14267+ return (stbir_real)numer * ( in_input_pixels ? scale_info->tap_scale_in : scale_info->tap_scale_out );
14268+ }
14269+ if ( in_input_pixels )
14270+ return ( ( (stbir_real)o + (stbir_real)0.5 + scale_info->pixel_shift ) * scale_info->inv_scale ) - ( (stbir_real)i + (stbir_real)0.5 );
14271+ return ( (stbir_real)o + (stbir_real)0.5 ) - ( ( (stbir_real)i + (stbir_real)0.5 ) * scale_info->scale - scale_info->pixel_shift );
14272+}
14273+
14274+// the kernel's weight at x (in filter pixels): through the new form directly, or an old float kernel's floats
14275+static stbir_real stbir__kernel_weight( stbir__kernel_callback_new * kernel_new, stbir__kernel_callback * kernel_old, stbir_real x, stbir_real scale, void * user_data )
14276+{
14277+ if ( kernel_new )
14278+ return kernel_new( x, scale, user_data );
14279+ return (stbir_real) kernel_old( (float) x, (float) scale, user_data );
14280+}
14281+
14282+static void stbir__exact_pixel_range( stbir_int64 numer, stbir_int64 den, stbir_real den_reciprocal, stbir_real support, int * first, int * last )
14283+{
14284+ int q = (int)( numer / den );
14285+ stbir_real f = (stbir_real)( numer % den ) * den_reciprocal; // ( -1, 1 ): the same sign as numer
14286+ *first = q + stbir__floor_to_int( f - support + (stbir_real)0.5 + stbir__tie_epsilon );
14287+ *last = q + stbir__floor_to_int( f + support - (stbir_real)0.5 + stbir__tie_epsilon );
14288+}
14289+
14290+static void stbir__exact_output_range( stbir_int64 numer, stbir_int64 den, stbir_real den_reciprocal, stbir_real support, int * first, int * last )
14291+{
14292+ int q = (int)( numer / den );
14293+ stbir_real f = (stbir_real)( numer % den ) * den_reciprocal;
14294+ *first = q + stbir__floor_to_int( f - support - (stbir_real)0.5 - stbir__tie_epsilon ) + 1; // ceil, a tie exactly on it included
14295+ *last = q + stbir__floor_to_int( f + support - (stbir_real)0.5 - stbir__tie_epsilon ); // a tie exactly on it excluded
14296+}
14297+
14298+// The input pixels within the filter's support of output pixel o: out_filter_radius is the support in output pixels for
14299+// the real chain, support the same in input pixels (the filter's own, for upsamples) for the exact positions - then
14300+// the center is the one stbir__tap_offset weights from.
14301+static void stbir__output_in_pixel_range( stbir__scale_info const * scale_info, int o, stbir_real out_filter_radius, stbir_real support, stbir_edge edge, int * first_pixel, int * last_pixel )
14302+{
14303+ if ( scale_info->positions_exact )
14304+ {
14305+ stbir_int64 numer = ( 2 * (stbir_int64)o + 1 + 2 * (stbir_int64)scale_info->shift_pixels ) * (stbir_int64)scale_info->scale_denominator; // output o's center in input pixels, over 2N
14306+ int first, last;
14307+ stbir__exact_pixel_range( numer, 2 * (stbir_int64)scale_info->scale_numerator, scale_info->tap_scale_in, support, &first, &last );
14308+ stbir__clip_in_pixel_range( first, last, scale_info->input_full_size, edge, first_pixel, last_pixel );
14309+ }
14310+ else
14311+ stbir__calculate_in_pixel_range( first_pixel, last_pixel, (stbir_real)o + (stbir_real)0.5, out_filter_radius, scale_info->inv_scale, scale_info->pixel_shift, scale_info->input_full_size, edge );
14312+}
14313+
14314+static void stbir__calculate_coefficients_for_gather_upsample( stbir_real out_filter_radius, stbir_real support, stbir__kernel_callback_new * kernel_new, stbir__kernel_callback * kernel_old, stbir__scale_info * scale_info, int num_contributors, stbir__contributors* contributors, float* coefficient_group, int coefficient_width, stbir_edge edge, void * user_data )
14315+{
14316+ int n, end;
14317+ int numerator = scale_info->scale_numerator;
14318+ int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < num_contributors ) );
14319+
14320+ // Looping through out pixels
14321+ end = num_contributors; if ( polyphase ) end = numerator;
14322+ for (n = 0; n < end; n++)
14323+ {
14324+ int i;
14325+ int last_non_zero;
14326+ int in_first_pixel, in_last_pixel;
14327+
14328+ stbir__output_in_pixel_range( scale_info, n, out_filter_radius, support, edge, &in_first_pixel, &in_last_pixel );
14329+
14330+ // make sure we never generate a range larger than our precalculated coeff width
14331+ // this only happens in point sample mode, but it's a good safe thing to do anyway
14332+ if ( ( in_last_pixel - in_first_pixel + 1 ) > coefficient_width )
14333+ in_last_pixel = in_first_pixel + coefficient_width - 1;
14334+
14335+ last_non_zero = -1;
14336+ for (i = 0; i <= in_last_pixel - in_first_pixel; i++)
14337+ {
14338+ stbir_real weight = stbir__kernel_weight( kernel_new, kernel_old, stbir__tap_offset( scale_info, n, i + in_first_pixel, 1 ), scale_info->inv_scale, user_data );
14339+ float coeff = (float) weight;
14340+
14341+ if ( ( weight < stbir__too_small_coefficient ) && ( weight > -stbir__too_small_coefficient ) )
14342+ {
14343+ if ( i == 0 ) // if we're at the front, just eat zero contributors
14344+ {
14345+ STBIR_ASSERT ( ( in_last_pixel - in_first_pixel ) != 0 ); // there should be at least one contrib
14346+ ++in_first_pixel;
14347+ i--;
14348+ continue;
14349+ }
14350+ coeff = 0; // make sure is fully zero (should keep denormals away)
14351+ }
14352+ else
14353+ last_non_zero = i;
14354+
14355+ coefficient_group[i] = coeff;
14356+ }
14357+
14358+ in_last_pixel = last_non_zero+in_first_pixel; // kills trailing zeros
14359+ contributors->n0 = in_first_pixel;
14360+ contributors->n1 = in_last_pixel;
14361+
14362+ STBIR_ASSERT(contributors->n1 >= contributors->n0);
14363+
14364+ ++contributors;
14365+ coefficient_group += coefficient_width;
14366+ }
14367+}
14368+
14369+static void stbir__insert_coeff( stbir__contributors * contribs, float * coeffs, int new_pixel, float new_coeff, int max_width )
14370+{
14371+ if ( contribs->n1 < contribs->n0 ) // this first clause should never happen, but handle in case
14372+ {
14373+ contribs->n0 = contribs->n1 = new_pixel;
14374+ coeffs[0] = new_coeff;
14375+ }
14376+ else if ( new_pixel <= contribs->n1 ) // before the end
14377+ {
14378+ if ( new_pixel < contribs->n0 ) // before the front?
14379+ {
14380+ if ( ( contribs->n1 - new_pixel + 1 ) <= max_width )
14381+ {
14382+ int j, o = contribs->n0 - new_pixel;
14383+ for ( j = contribs->n1 - contribs->n0 ; j >= 0 ; j-- )
14384+ coeffs[ j + o ] = coeffs[ j ];
14385+ for ( j = 1 ; j < o ; j++ )
14386+ coeffs[ j ] = 0;
14387+ coeffs[ 0 ] = new_coeff;
14388+ contribs->n0 = new_pixel;
14389+ }
14390+ }
14391+ else
14392+ {
14393+ // add new weight to existing coeff if already there
14394+ coeffs[ new_pixel - contribs->n0 ] += new_coeff;
14395+ }
14396+ }
14397+ else
14398+ {
14399+ if ( ( new_pixel - contribs->n0 + 1 ) <= max_width )
14400+ {
14401+ int j, e = new_pixel - contribs->n0;
14402+ for( j = ( contribs->n1 - contribs->n0 ) + 1 ; j < e ; j++ ) // clear in-betweens coeffs if there are any
14403+ coeffs[j] = 0;
14404+
14405+ coeffs[ e ] = new_coeff;
14406+ contribs->n1 = new_pixel;
14407+ }
14408+ }
14409+}
14410+
14411+// The output pixels within in_pixels_radius (input pixels) of an input pixel's center.
14412+static void stbir__calculate_out_pixel_range( int * first_pixel, int * last_pixel, stbir_real in_pixel_center, stbir_real in_pixels_radius, stbir_real scale, stbir_real out_shift, int out_size )
14413+{
14414+ stbir_real out_pixel_influence_lowerbound = ( in_pixel_center - in_pixels_radius ) * scale - out_shift;
14415+ stbir_real out_pixel_influence_upperbound = ( in_pixel_center + in_pixels_radius ) * scale - out_shift;
14416+ stbir__clip_out_pixel_range( stbir__floor_to_int( out_pixel_influence_lowerbound - (stbir_real)0.5 - stbir__tie_epsilon ) + 1, stbir__floor_to_int( out_pixel_influence_upperbound - (stbir_real)0.5 - stbir__tie_epsilon ), out_size, first_pixel, last_pixel );
14417+}
14418+
14419+// The output pixels within the filter's support of input pixel i, one pixel at a time - the same ways as
14420+// stbir__output_in_pixel_range: in_pixels_radius is the support in input pixels for the real chain, support the
14421+// same in output pixels for the exact positions.
14422+static void stbir__input_out_pixel_range( stbir__scale_info const * scale_info, int i, stbir_real in_pixels_radius, stbir_real support, int * first_pixel, int * last_pixel )
14423+{
14424+ if ( scale_info->positions_exact )
14425+ {
14426+ stbir_int64 numer = ( 2 * (stbir_int64)i + 1 ) * (stbir_int64)scale_info->scale_numerator - 2 * (stbir_int64)scale_info->shift_pixels * (stbir_int64)scale_info->scale_denominator; // input i's center in output pixels, over 2D
14427+ int first, last;
14428+ stbir__exact_output_range( numer, 2 * (stbir_int64)scale_info->scale_denominator, scale_info->tap_scale_out, support, &first, &last );
14429+ stbir__clip_out_pixel_range( first, last, scale_info->output_sub_size, first_pixel, last_pixel );
14430+ }
14431+ else
14432+ stbir__calculate_out_pixel_range( first_pixel, last_pixel, (stbir_real)i + (stbir_real)0.5, in_pixels_radius, scale_info->scale, scale_info->pixel_shift, scale_info->output_sub_size );
14433+}
14434+
14435+static void stbir__calculate_coefficients_for_gather_downsample( int start, int end, stbir_real in_pixels_radius, stbir_real support, stbir__kernel_callback_new * kernel_new, stbir__kernel_callback * kernel_old, stbir__scale_info * scale_info, int coefficient_width, int num_contributors, stbir__contributors * contributors, float * coefficient_group, void * user_data )
14436+{
14437+ int in_pixel;
14438+ int i;
14439+ int first_out_inited = -1;
14440+ int out_size = scale_info->output_sub_size;
14441+ int numerator = scale_info->scale_numerator;
14442+ int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < out_size ) );
14443+
14444+ STBIR__UNUSED(num_contributors);
14445+
14446+ // Loop through the input pixels
14447+ for (in_pixel = start; in_pixel < end; in_pixel++)
14448+ {
14449+ int out_first_pixel, out_last_pixel;
14450+
14451+ stbir__input_out_pixel_range( scale_info, in_pixel, in_pixels_radius, support, &out_first_pixel, &out_last_pixel );
14452+
14453+ if ( out_first_pixel > out_last_pixel )
14454+ continue;
14455+
14456+ // clamp or exit if we are using polyphase filtering, and the limit is up
14457+ if ( polyphase )
14458+ {
14459+ // when polyphase, you only have to do coeffs up to the numerator count
14460+ if ( out_first_pixel >= numerator )
14461+ break;
14462+
14463+ // don't do any extra work, clamp last pixel at numerator too
14464+ if ( out_last_pixel >= numerator )
14465+ out_last_pixel = numerator - 1;
14466+ }
14467+
14468+ for (i = 0; i <= out_last_pixel - out_first_pixel; i++)
14469+ {
14470+ stbir_real weight = stbir__kernel_weight( kernel_new, kernel_old, stbir__tap_offset( scale_info, i + out_first_pixel, in_pixel, 0 ), scale_info->scale, user_data );
14471+ float coeff = 0.0f;
14472+
14473+ if ( ( weight >= stbir__too_small_coefficient ) || ( weight <= -stbir__too_small_coefficient ) )
14474+ coeff = (float) ( weight * scale_info->scale );
14475+
14476+ {
14477+ int out = i + out_first_pixel;
14478+ float * coeffs = coefficient_group + (size_t)out * coefficient_width;
14479+ stbir__contributors * contribs = contributors + out;
14480+
14481+ // is this the first time this output pixel has been seen? Init it.
14482+ if ( out > first_out_inited )
14483+ {
14484+ STBIR_ASSERT( out == ( first_out_inited + 1 ) ); // ensure we have only advanced one at time
14485+ first_out_inited = out;
14486+ contribs->n0 = in_pixel;
14487+ contribs->n1 = in_pixel;
14488+ coeffs[0] = coeff;
14489+ }
14490+ else
14491+ {
14492+ // insert on end (always in order)
14493+ if ( coeffs[0] == 0.0f ) // if the first coefficient is zero, then zap it for this coeff
14494+ {
14495+ STBIR_ASSERT( ( in_pixel - contribs->n0 ) == 1 ); // ensure that when we zap, we're at the 2nd pos
14496+ contribs->n0 = in_pixel;
14497+ }
14498+ contribs->n1 = in_pixel;
14499+ STBIR_ASSERT( ( in_pixel - contribs->n0 ) < coefficient_width );
14500+ coeffs[in_pixel - contribs->n0] = coeff;
14501+ }
14502+ }
14503+ }
14504+ }
14505+}
14506+
14507+// fold a contributor's out of range taps back into the image for the edge mode, trim its trailing zero weights,
14508+// and fold its extents into lowest/highest/widest
14509+static stbir__inline void stbir__fold_and_trim_contributor( stbir_edge edge, stbir__contributors * contributor, float * coefficient_group, int coefficient_width, int input_size, stbir__filter_extent_info * extents )
14510+{
14511+ int input_last_pixel = input_size - 1;
14512+ int i;
14513+
14514+ if ( edge == STBIR_EDGE_ZERO )
14515+ {
14516+ // In zero edge mode, just remove the out of bounds pixels completely (their weights are accounted for now).
14517+ if ( contributor->n1 > input_last_pixel )
14518+ contributor->n1 = input_last_pixel;
14519+
14520+ if ( contributor->n0 < 0 )
14521+ {
14522+ int skip = -contributor->n0;
14523+ int remaining;
14524+
14525+ contributor->n0 = 0;
14526+
14527+ // Move the weights down.
14528+ remaining = contributor->n1 - contributor->n0 + 1;
14529+ for (i = 0; i < remaining; i++)
14530+ coefficient_group[i] = coefficient_group[i + skip];
14531+ }
14532+ }
14533+ else if ( ( edge == STBIR_EDGE_CLAMP ) || ( edge == STBIR_EDGE_REFLECT ) )
14534+ {
14535+ // For clamp and reflect, calculate the true in bounds pixel (based on the edge type) and add the weight to it.
14536+
14537+ // Right hand side first.
14538+ if ( contributor->n1 > input_last_pixel )
14539+ {
14540+ int first = contributor->n0;
14541+ int last = contributor->n1;
14542+ contributor->n1 = input_last_pixel;
14543+ for (i = input_size; i <= last; i++)
14544+ stbir__insert_coeff( contributor, coefficient_group, stbir__edge_wrap_slow[edge]( i, input_size ), coefficient_group[i - first], coefficient_width );
14545+ }
14546+
14547+ // Then the left.
14548+ if ( contributor->n0 < 0 )
14549+ {
14550+ int first = contributor->n0;
14551+ float * coeff = coefficient_group - ( first + 1 ); // the weight of pixel -1
14552+ float first_coeff;
14553+
14554+ // Reinsert the weights at their reflected or clamped pixel (insert accumulates if the pixel is already there).
14555+ for (i = -1; i > first; i--)
14556+ stbir__insert_coeff( contributor, coefficient_group, stbir__edge_wrap_slow[edge]( i, input_size ), *coeff--, coefficient_width );
14557+
14558+ // Save the first one, since we didn't do it yet (i == n0) - there might be too many coeffs to hold before we shrink.
14559+ first_coeff = coeff[0];
14560+
14561+ // Slide all the coeffs down (we accumulated them into the positive pixels) and reset the first pixel.
14562+ contributor->n0 = 0;
14563+ for (i = 0; i <= contributor->n1; i++)
14564+ coefficient_group[i] = coefficient_group[i - first];
14565+
14566+ // Now that we have shrunk the contributor, we can insert the first one safely.
14567+ stbir__insert_coeff( contributor, coefficient_group, stbir__edge_wrap_slow[edge]( first, input_size ), first_coeff, coefficient_width );
14568+ }
14569+ }
14570+
14571+ // Trim trailing zero coefficients, re-zero the unused tail, and widen the extents to this contributor.
14572+ if ( contributor->n0 <= contributor->n1 )
14573+ {
14574+ int count = contributor->n1 - contributor->n0 + 1;
14575+ while ( count && ( coefficient_group[count - 1] == 0.0f ) )
14576+ --count;
14577+
14578+ contributor->n1 = contributor->n0 + count - 1;
14579+
14580+ if ( count > 0 )
14581+ {
14582+ if ( contributor->n0 < extents->lowest )
14583+ extents->lowest = contributor->n0;
14584+ if ( contributor->n1 > extents->highest )
14585+ extents->highest = contributor->n1;
14586+ if ( count > extents->widest )
14587+ extents->widest = count;
14588+ }
14589+
14590+ for (i = count; i < coefficient_width; i++)
14591+ coefficient_group[i] = 0.0f;
14592+ }
14593+}
14594+
14595+static void stbir__cleanup_gathered_coefficients( stbir_edge edge, stbir__filter_extent_info* filter_info, stbir__scale_info * scale_info, int num_contributors, stbir__contributors* contributors, float * coefficients, int coefficient_width )
14596+{
14597+ int input_size = scale_info->input_full_size;
14598+ int numerator = scale_info->scale_numerator;
14599+ int denominator = scale_info->scale_denominator;
14600+ int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < num_contributors ) );
14601+ int calculated = polyphase ? numerator : num_contributors; // when polyphase, only the first numerator outputs were calculated
14602+ stbir_real too_small_sum = stbir__too_small_coefficient_sum_for( scale_info );
14603+ int n, i;
14604+
14605+ // Make the sum of each output's coefficients 1.
14606+ for (n = 0; n < calculated; n++)
14607+ {
14608+ stbir__contributors * contributor = contributors + n;
14609+ float * coefficient_group = coefficients + (size_t)n * coefficient_width;
14610+ stbir_real total_filter = 0;
14611+ int last = contributor->n1 - contributor->n0;
14612+
14613+ for (i = 0; i <= last; i++)
14614+ {
14615+ total_filter += (stbir_real) coefficient_group[i];
14616+ STBIR_ASSERT( ( coefficient_group[i] >= -2.0f ) && ( coefficient_group[i] <= 2.0f ) ); // check for wonky weights
14617+ }
14618+
14619+ if ( ( total_filter < too_small_sum ) && ( total_filter > -too_small_sum ) )
14620+ {
14621+ // All coeffs are extremely small, just zero it.
14622+ contributor->n1 = contributor->n0;
14623+ coefficient_group[0] = 0.0f;
14624+ }
14625+ else if ( ( total_filter < ( (stbir_real)1.0 - stbir__renorm_tolerance ) ) || ( total_filter > ( (stbir_real)1.0 + stbir__renorm_tolerance ) ) )
14626+ {
14627+ stbir_real filter_scale = (stbir_real)1.0 / total_filter;
14628+ for (i = 0; i <= last; i++)
14629+ coefficient_group[i] = (float) ( coefficient_group[i] * filter_scale );
14630+ }
14631+ }
14632+
14633+ // Polyphase: the rest of the outputs are the first numerator's, moved along by the denominator once per period.
14634+ if ( polyphase )
14635+ {
14636+ for (n = numerator; n < num_contributors; n++)
14637+ {
14638+ contributors[n].n0 = contributors[n - numerator].n0 + denominator;
14639+ contributors[n].n1 = contributors[n - numerator].n1 + denominator;
14640+ }
14641+ stbir_overlapping_memcpy( coefficients + (size_t)numerator * coefficient_width, coefficients, (size_t)( num_contributors - numerator ) * coefficient_width );
14642+ }
14643+
14644+ filter_info->lowest = 0x7fffffff;
14645+ filter_info->highest = -0x7fffffff;
14646+ filter_info->widest = -1;
14647+ for (n = 0; n < num_contributors; n++)
14648+ stbir__fold_and_trim_contributor( edge, contributors + n, coefficients + (size_t)n * coefficient_width, coefficient_width, input_size, filter_info );
14649+}
14650+
14651+// how many input pixels the horizontal kernel that gets selected for this widest will read
14652+static int stbir__horizontal_read_extent( stbir__contributors const * contribs, int widest )
14653+{
14654+ int mod, extent;
14655+
14656+ if ( widest <= 17 )
14657+ return widest;
14658+
14659+ mod = widest & 3;
14660+ extent = ( ( ( contribs->n1 - contribs->n0 + 1 ) - mod + 3 ) & ~3 ) + mod;
14661+ if ( extent < ( 8 + mod ) )
14662+ extent = 8 + mod;
14663+ return extent;
14664+}
14665+
14666+// the SIMD kernels load a remnant of three coefficients as four, so widths of 4n+3 get a fourth, zero coefficient
14667+#define STBIR__COEFFICIENT_PAD( widest ) ( ( (widest) & 3 ) == 3 )
14668+#define STBIR__COLUMN_ENTRY_FLOATS( widest ) ( 2 + (widest) + STBIR__COEFFICIENT_PAD( widest ) ) // the entry's size in floats
14669+
14670+// Repack the coefficient groups from coefficient_width floats each to the run time layout: a stbir__column_entry,
14671+// two ints (filled in by stbir__fill_column_entries) followed by widest coefficients.
14672+static void stbir__pack_column_entries( int num_entries, float * coefficients, int coefficient_width, int widest )
14673+{
14674+ int entry_floats = STBIR__COLUMN_ENTRY_FLOATS( widest );
14675+ int i, j;
14676+
14677+ for (i = 0; i < num_entries; i++)
14678+ {
14679+ // When the entries grow they move up, so do the last one first; otherwise they move down (or stay), first one first.
14680+ int n = ( entry_floats > coefficient_width ) ? ( num_entries - 1 - i ) : i;
14681+ float * coefficient_group = coefficients + (size_t)n * coefficient_width;
14682+ float * entry_coefficients = coefficients + (size_t)n * entry_floats + 2;
14683+
14684+ if ( entry_coefficients > coefficient_group )
14685+ {
14686+ for (j = widest - 1; j >= 0; j--)
14687+ entry_coefficients[j] = coefficient_group[j];
14688+ }
14689+ else
14690+ {
14691+ for (j = 0; j < widest; j++)
14692+ entry_coefficients[j] = coefficient_group[j];
14693+ }
14694+
14695+ // The kernels that load one float past the coefficients see a zero.
14696+ if ( STBIR__COEFFICIENT_PAD( widest ) )
14697+ entry_coefficients[widest] = 0.0f;
14698+ }
14699+}
14700+
14701+// the horizontal kernels read a fixed number of pixels from each contributor's n0 (see stbir__horizontal_read_extent),
14702+// so contributors near the right edge get moved back (zero weights in front) to keep those reads inside the
14703+// decoded range.
14704+static void stbir__clip_contributor_reads( int num_contributors, stbir__contributors* contributors, float * coefficients, int stride, int widest, int row0, int row1 )
14705+{
14706+ int row_end = row1 + 1;
14707+ STBIR__UNUSED( row0 ); // only used in an assert
14708+
14709+ // the fixed width kernels (up to 17 taps) read exactly widest pixels from n0 whatever a contributor's own width is,
14710+ // and the generic ones read in fours (stbir__horizontal_read_extent). So we make sure we never read outside the
14711+ // decode buffer, by possibly moving the sample area back into the scanline, and putting zero weights first.
14712+ // we start on the right edge and check until we're well past the possible
14713+ // clip area (2*widest).
14714+ {
14715+ stbir__contributors * contribs = contributors + num_contributors - 1;
14716+ float * coeffs = coefficients + (size_t)stride * ( num_contributors - 1 );
14717+
14718+ // go until no chance of clipping (this is usually less than 8 lops)
14719+ while ( ( contribs >= contributors ) && ( ( contribs->n0 + widest*2 ) >= row_end ) )
14720+ {
14721+ // might we clip??
14722+ if ( ( contribs->n0 + widest ) > row_end )
14723+ {
14724+ // past 17 taps the generic loops run, and they read to this contributor's own width rounded up to a four,
14725+ // not a fixed widest amount - so calculate this
14726+ int stop_range = stbir__horizontal_read_extent( contribs, widest );
14727+
14728+ // now see if we still clip with the refined range
14729+ if ( ( contribs->n0 + stop_range ) > row_end )
14730+ {
14731+ int new_n0 = row_end - stop_range;
14732+ int num = contribs->n1 - contribs->n0 + 1;
14733+ int backup = contribs->n0 - new_n0;
14734+ float * from_co = coeffs + num - 1;
14735+ float * to_co = from_co + backup;
14736+
14737+ STBIR_ASSERT( ( new_n0 >= row0 ) && ( new_n0 < contribs->n0 ) );
14738+
14739+ // move the coeffs over
14740+ while( num )
14741+ {
14742+ *to_co-- = *from_co--;
14743+ --num;
14744+ }
14745+ // zero new positions
14746+ while ( to_co >= coeffs )
14747+ *to_co-- = 0;
14748+ // set new start point (backing up by less than the mod-4 slack keeps the read extent the same, so this can't clip again)
14749+ contribs->n0 = new_n0;
14750+ STBIR_ASSERT( ( contribs->n0 + stbir__horizontal_read_extent( contribs, widest ) ) <= row_end );
14751+ }
14752+ }
14753+ --contribs;
14754+ coeffs -= stride;
14755+ }
14756+ }
14757+}
14758+
14759+// pack the coefficient sets into column entries and clip the edge reads
14760+static int stbir__pack_coefficients( int num_contributors, stbir__contributors* contributors, float * coefficients, int coefficient_width, int widest, int row0, int row1 )
14761+{
14762+ stbir__pack_column_entries( num_contributors, coefficients, coefficient_width, widest );
14763+ stbir__clip_contributor_reads( num_contributors, contributors, coefficients + 2, STBIR__COLUMN_ENTRY_FLOATS( widest ), widest, row0, row1 );
14764+ return widest;
14765+}
14766+
14767+// For polyphase resizing (rational scale resizing), there is a ton of coefficent sharing across the
14768+// horizontal scanline. We store the data like this (you need the pro and epi for clamp modes):
14769+// [ prologue scanline coeffs ][ big run of repeating coeffs ][ epilogue scanline coeffs ]
14770+
14771+// Bounds on how many outputs at each edge have taps outside the image (before folding), from the filter geometry.
14772+static void stbir__polyphase_edge_bounds( stbir__sampler const * samp, void * user_data, int * out_prologue, int * out_epilogue )
14773+{
14774+ stbir__scale_info const * scale_info = &samp->scale_info;
14775+ int num_contributors = samp->num_contributors;
14776+ stbir_real in_pixels_radius;
14777+ int prologue = 0;
14778+ int epilogue = 0;
14779+ int n;
14780+
14781+ // the reach of an output's taps, plus how far an approximated fraction's copies can drift from these real positions
14782+ in_pixels_radius = stbir__filter_radius_in_input_pixels( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, scale_info, user_data ) + (stbir_real)scale_info->approximated_drift;
14783+
14784+ // Nothing gets folded with wrap.
14785+ if ( samp->edge != STBIR_EDGE_WRAP )
14786+ {
14787+ // Looping through out pixels from the left edge, until one reads nothing before the image
14788+ for (n = 0; n < num_contributors; n++)
14789+ {
14790+ stbir_real in_first_pixel = ( (stbir_real)n + (stbir_real)0.5 + scale_info->pixel_shift ) * scale_info->inv_scale - in_pixels_radius;
14791+ if ( stbir__floor_to_int( in_first_pixel + (stbir_real)0.5 ) >= 0 )
14792+ break;
14793+ ++prologue;
14794+ }
14795+
14796+ // And from the right edge, until one reads nothing past it
14797+ for (n = num_contributors - 1; n >= 0; n--)
14798+ {
14799+ stbir_real in_last_pixel = ( (stbir_real)n + (stbir_real)0.5 + scale_info->pixel_shift ) * scale_info->inv_scale + in_pixels_radius;
14800+ if ( stbir__floor_to_int( in_last_pixel - (stbir_real)0.5 ) < scale_info->input_full_size )
14801+ break;
14802+ ++epilogue;
14803+ }
14804+
14805+ // A pixel of slack each way: this is the real chain, the generators' positions can be exact.
14806+ ++prologue;
14807+ ++epilogue;
14808+ }
14809+
14810+ *out_prologue = prologue;
14811+ *out_epilogue = epilogue;
14812+}
14813+
14814+// for polyphase (rational) scales, we want to unroll the sharing a bit just so we aren't
14815+// constantly breaking out of a tight loop to back up the coeff sharing.
14816+#define STBIR__POLYPHASE_MIN_BLOCK 256
14817+
14818+// decide whether this (gather) sampler shares its polyphase coefficient sets, and size its tables accordingly.
14819+// called before the allocation pass, so it can only use the sampler's geometry.
14820+static void stbir__plan_polyphase_sharing( stbir__sampler * samp, int reads_get_clipped, void * user_data )
14821+{
14822+ int num_contributors = samp->num_contributors;
14823+ int numerator = (int) samp->scale_info.scale_numerator;
14824+ int prologue, epilogue, period, repeating;
14825+
14826+ samp->shared_period = 0;
14827+ samp->shared_prologue = 0;
14828+ samp->shared_epilogue = 0;
14829+
14830+ if ( ( !samp->is_gather ) || ( !samp->scale_info.scale_is_rational ) || ( numerator >= num_contributors ) )
14831+ return;
14832+
14833+ stbir__polyphase_edge_bounds( samp, user_data, &prologue, &epilogue );
14834+
14835+ // the prologue is whole periods, so an output's phase is its index mod numerator
14836+ prologue = ( ( prologue + numerator - 1 ) / numerator ) * numerator;
14837+
14838+ // the horizontal's clip (stbir__clip_contributor_reads) can also move outputs within 2*widest input pixels of the
14839+ // right edge: 2*widest*scale outputs, bounded with the coefficient width since widest isn't known yet
14840+ if ( reads_get_clipped )
14841+ epilogue += (int)( (stbir_real)2.0 * (stbir_real)samp->coefficient_width * samp->scale_info.scale ) + 2;
14842+
14843+ // the block is one period of sets (one set at period 1, run with an entry stride of zero). the horizontal walks a
14844+ // row through it, so its block is padded out to keep the kernel calls long; the vertical fetches a set per row by index
14845+ period = numerator;
14846+ if ( ( numerator > 1 ) && ( reads_get_clipped ) )
14847+ {
14848+ while ( period < STBIR__POLYPHASE_MIN_BLOCK )
14849+ period += numerator;
14850+ }
14851+
14852+ // not enough repeating outputs to be worth it
14853+ repeating = num_contributors - prologue - epilogue;
14854+ if ( repeating < period )
14855+ return;
14856+
14857+ samp->shared_period = period;
14858+ samp->shared_prologue = prologue;
14859+ samp->shared_epilogue = epilogue;
14860+ samp->coefficients_size = (size_t)( prologue + period + epilogue ) * STBIR__COLUMN_ENTRY_FLOATS( samp->coefficient_width ) * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING;
14861+ samp->contributors_size = (size_t)( prologue + period + epilogue + 1 ) * sizeof(stbir__contributors); // one per coefficient set (build time only)
14862+}
14863+
14864+// the shared layout keeps contributors for just the outputs that have their own coefficient sets, in set order:
14865+// [ P prologue outputs ][ block outputs ][ E epilogue outputs ].
14866+static void stbir__shared_output_range( stbir__sampler const * samp, int x, int * n0, int * n1 )
14867+{
14868+ int prologue = samp->shared_prologue;
14869+ int period = samp->shared_period;
14870+ int epilogue = samp->shared_epilogue;
14871+ int num_contributors = samp->num_contributors;
14872+ stbir__contributors const * contributor;
14873+ int shift = 0;
14874+
14875+ if ( x >= num_contributors - epilogue )
14876+ contributor = samp->contributors + prologue + period + ( x - ( num_contributors - epilogue ) );
14877+ else if ( x >= prologue )
14878+ {
14879+ contributor = samp->contributors + prologue + ( ( x - prologue ) % period );
14880+ shift = ( ( x - prologue ) / period ) * ( period / (int) samp->scale_info.scale_numerator ) * (int) samp->scale_info.scale_denominator;
14881+ }
14882+ else
14883+ contributor = samp->contributors + x;
14884+
14885+ *n0 = contributor->n0 + shift;
14886+ *n1 = contributor->n1 + shift;
14887+}
14888+
14889+// n0/n1 of output x for either layout
14890+static void stbir__output_range( stbir__sampler const * samp, int x, int * n0, int * n1 )
14891+{
14892+ if ( samp->shared_period )
14893+ stbir__shared_output_range( samp, x, n0, n1 );
14894+ else
14895+ {
14896+ *n0 = samp->contributors[ x ].n0;
14897+ *n1 = samp->contributors[ x ].n1;
14898+ }
14899+}
14900+
14901+// The shared layout's version of stbir__cleanup_gathered_coefficients.
14902+static void stbir__cleanup_shared_coefficients( stbir__sampler * samp )
14903+{
14904+ stbir_edge edge = samp->edge;
14905+ stbir__scale_info const * scale_info = &samp->scale_info;
14906+ int input_size = scale_info->input_full_size;
14907+ int num_contributors = samp->num_contributors;
14908+ int coefficient_width = samp->coefficient_width;
14909+ int numerator = (int) scale_info->scale_numerator;
14910+ int denominator = (int) scale_info->scale_denominator;
14911+ int prologue = samp->shared_prologue;
14912+ int period = samp->shared_period;
14913+ int epilogue = samp->shared_epilogue;
14914+ stbir__contributors * contributors = samp->contributors;
14915+ float * coefficients = samp->coefficients;
14916+ float * phase_coefficients = coefficients + (size_t)prologue * coefficient_width;
14917+ stbir__filter_extent_info * extents = &samp->extent_info;
14918+ stbir_real too_small_sum = stbir__too_small_coefficient_sum_for( &samp->scale_info );
14919+ int n, i;
14920+
14921+ // Make the sum of each phase's coefficients 1 (as the plain path does for its first numerator outputs).
14922+ for (n = 0; n < numerator; n++)
14923+ {
14924+ stbir__contributors * contributor = contributors + n;
14925+ float * coefficient_group = phase_coefficients + (size_t)n * coefficient_width;
14926+ stbir_real total_filter = 0;
14927+ int last = contributor->n1 - contributor->n0;
14928+
14929+ for (i = 0; i <= last; i++)
14930+ {
14931+ total_filter += (stbir_real) coefficient_group[i];
14932+ STBIR_ASSERT( ( coefficient_group[i] >= -2.0f ) && ( coefficient_group[i] <= 2.0f ) ); // check for wonky weights
14933+ }
14934+
14935+ if ( ( total_filter < too_small_sum ) && ( total_filter > -too_small_sum ) )
14936+ {
14937+ // All coeffs are extremely small, just zero it.
14938+ contributor->n1 = contributor->n0;
14939+ coefficient_group[0] = 0.0f;
14940+ }
14941+ else if ( ( total_filter < ( (stbir_real)1.0 - stbir__renorm_tolerance ) ) || ( total_filter > ( (stbir_real)1.0 + stbir__renorm_tolerance ) ) )
14942+ {
14943+ stbir_real filter_scale = (stbir_real)1.0 / total_filter;
14944+ for (i = 0; i <= last; i++)
14945+ coefficient_group[i] = (float) ( coefficient_group[i] * filter_scale );
14946+ }
14947+ }
14948+
14949+ // The contributors for the prologue and period outputs (0..prologue+period-1) are the phase's, moved along by the
14950+ // denominator once per period. The epilogue outputs (the last epilogue of the image) follow them.
14951+ for (n = numerator; n < prologue + period; n++)
14952+ {
14953+ contributors[n].n0 = contributors[n - numerator].n0 + denominator;
14954+ contributors[n].n1 = contributors[n - numerator].n1 + denominator;
14955+ }
14956+ for (n = 0; n < epilogue; n++)
14957+ {
14958+ int out_pixel = num_contributors - epilogue + n;
14959+ contributors[prologue + period + n].n0 = contributors[out_pixel % numerator].n0 + ( out_pixel / numerator ) * denominator;
14960+ contributors[prologue + period + n].n1 = contributors[out_pixel % numerator].n1 + ( out_pixel / numerator ) * denominator;
14961+ }
14962+
14963+ // The period is the phases repeated; the prologue and epilogue groups start out as copies of their phase.
14964+ for (n = numerator; n < period; n++)
14965+ STBIR_MEMCPY( phase_coefficients + (size_t)n * coefficient_width, phase_coefficients + (size_t)( n % numerator ) * coefficient_width, coefficient_width * sizeof(float) );
14966+ for (n = 0; n < prologue; n++)
14967+ STBIR_MEMCPY( coefficients + (size_t)n * coefficient_width, phase_coefficients + (size_t)( n % numerator ) * coefficient_width, coefficient_width * sizeof(float) );
14968+ for (n = 0; n < epilogue; n++)
14969+ STBIR_MEMCPY( coefficients + (size_t)( prologue + period + n ) * coefficient_width, phase_coefficients + (size_t)( ( num_contributors - epilogue + n ) % numerator ) * coefficient_width, coefficient_width * sizeof(float) );
14970+
14971+ // Fold and trim every group. The period's outputs are never at an edge (that is what the prologue and epilogue are
14972+ // for), so for them this is just the trim, which is the same for every copy of a phase - the period stays periodic.
14973+ extents->lowest = 0x7fffffff;
14974+ extents->highest = -0x7fffffff;
14975+ extents->widest = -1;
14976+ for (n = 0; n < prologue + period + epilogue; n++)
14977+ {
14978+ STBIR_ASSERT( ( n < prologue ) || ( n >= prologue + period ) || ( edge == STBIR_EDGE_WRAP ) || ( ( contributors[n].n0 >= 0 ) && ( contributors[n].n1 < input_size ) ) );
14979+ stbir__fold_and_trim_contributor( edge, contributors + n, coefficients + (size_t)n * coefficient_width, coefficient_width, input_size, extents );
14980+ }
14981+
14982+ // lowest and highest are over every output: the extremes are in the prologue and first period, and the last period
14983+ // and epilogue.
14984+ {
14985+ int n0, n1;
14986+ stbir__shared_output_range( samp, num_contributors - epilogue - 1, &n0, &n1 );
14987+ if ( n1 > extents->highest )
14988+ extents->highest = n1;
14989+ }
14990+}
14991+
14992+// the shared layout's version of stbir__pack_coefficients
14993+static int stbir__pack_shared_coefficients( stbir__sampler * samp, int row0, int row1 )
14994+{
14995+ int num_contributors = samp->num_contributors;
14996+ int widest = samp->extent_info.widest;
14997+ int entry_floats = STBIR__COLUMN_ENTRY_FLOATS( widest );
14998+ int prologue = samp->shared_prologue;
14999+ int period = samp->shared_period;
15000+ int epilogue = samp->shared_epilogue;
15001+ int n0, n1;
15002+
15003+ stbir__pack_column_entries( prologue + period + epilogue, samp->coefficients, samp->coefficient_width, widest );
15004+ stbir__clip_contributor_reads( epilogue, samp->contributors + prologue + period, samp->coefficients + (size_t)( prologue + period ) * entry_floats + 2, entry_floats, widest, row0, row1 );
15005+
15006+ // the clipping never needed to reach past the epilogue (the last periodic output is where it would have gone next)
15007+ stbir__shared_output_range( samp, num_contributors - epilogue - 1, &n0, &n1 );
15008+ STBIR_ASSERT( ( n0 + widest * 2 ) < row1 + 1 );
15009+ return widest;
15010+}
15011+
15012+// the packed column entry at an index (the entries are entry_floats apart)
15013+static stbir__column_entry * stbir__column_entry_at( stbir__sampler const * samp, int index )
15014+{
15015+ return (stbir__column_entry *)( samp->coefficients + (size_t)index * samp->entry_floats );
15016+}
15017+
15018+// fill in the column entries' advance and width from the build time contributors (after the entries have been
15019+// packed, and after the contributors have been read for the last time - the column slice table). returns 0 when
15020+// an advance doesn't fit the entry's int - a horizontal downsample of around 2^29:1 at four channels
15021+static int stbir__fill_column_entries( stbir__sampler * samp, int channels )
15022+{
15023+ stbir__contributors const * contributors = samp->contributors;
15024+ int num_entries = samp->shared_period ? ( samp->shared_prologue + samp->shared_period + samp->shared_epilogue ) : samp->num_contributors;
15025+ int i;
15026+
15027+ if ( samp->shared_period )
15028+ {
15029+ int prologue = samp->shared_prologue;
15030+ int period = samp->shared_period;
15031+ samp->shared_block_n0 = contributors[prologue].n0;
15032+ samp->shared_block_shift = ( period / (int) samp->scale_info.scale_numerator ) * (int) samp->scale_info.scale_denominator;
15033+ samp->shared_epilogue_n0 = contributors[prologue + period].n0;
15034+ }
15035+
15036+ // each entry says how far the decode pointer moves to the NEXT column. the block's last entry says how far to the
15037+ // block's first column one pass on (the wrap)
15038+ for (i = 0; i < num_entries; i++)
15039+ {
15040+ stbir__column_entry * entry = stbir__column_entry_at( samp, i );
15041+ int advance;
15042+
15043+ if ( ( samp->shared_period ) && ( i == samp->shared_prologue + samp->shared_period - 1 ) )
15044+ advance = ( samp->shared_block_n0 + samp->shared_block_shift ) - contributors[i].n0;
15045+ else
15046+ advance = ( i + 1 < num_entries ) ? ( contributors[i + 1].n0 - contributors[i].n0 ) : 0;
15047+
15048+ if ( ( (stbir_int64)advance * channels <= -0x7fffffff ) || ( (stbir_int64)advance * channels >= 0x7fffffff ) )
15049+ return 0; // error
15050+
15051+ entry->advance = advance * channels;
15052+ entry->width = contributors[i].n1 - contributors[i].n0 + 1;
15053+ }
15054+ return 1;
15055+}
15056+
15057+// which coefficient set (column entry) output x uses, for either layout
15058+static int stbir__output_entry_index( stbir__sampler const * samp, int x )
15059+{
15060+ if ( samp->shared_period )
15061+ {
15062+ int prologue = samp->shared_prologue;
15063+ int period = samp->shared_period;
15064+ int epilogue = samp->shared_epilogue;
15065+ int num_contributors = samp->num_contributors;
15066+
15067+ if ( x >= num_contributors - epilogue )
15068+ return prologue + period + ( x - ( num_contributors - epilogue ) );
15069+ if ( x >= prologue )
15070+ return prologue + ( ( x - prologue ) % period );
15071+ }
15072+ return x;
15073+}
15074+
15075+static float const * stbir__output_coefficients( stbir__sampler const * samp, int x )
15076+{
15077+ return samp->coefficients + (size_t)stbir__output_entry_index( samp, x ) * samp->coefficient_width;
15078+}
15079+
15080+// output x's packed column entry
15081+static stbir__column_entry const * stbir__output_column_entry( stbir__sampler const * samp, int x )
15082+{
15083+ return stbir__column_entry_at( samp, stbir__output_entry_index( samp, x ) );
15084+}
15085+
15086+// Vertical scatter functions - used when we are downsampling around 32:1 or more (horizontal is always gather)
15087+// advice-wise, if you are downsampling this amount, then you'd better off doing the downsample in a couple steps
15088+static void stbir__calculate_coefficients_scatter( stbir__kernel_callback_new * kernel_new, stbir__kernel_callback * kernel_old, stbir__scale_info const * scale_info, int in_pixel, int out_first_pixel, int out_last_pixel, stbir__contributors* contributor, float* coefficient_group, void * user_data )
15089+{
15090+ int i;
15091+
15092+ contributor->n0 = out_first_pixel;
15093+ contributor->n1 = out_last_pixel;
15094+
15095+ for (i = 0; i <= out_last_pixel - out_first_pixel; i++)
15096+ {
15097+ stbir_real weight = stbir__kernel_weight( kernel_new, kernel_old, stbir__tap_offset( scale_info, i + out_first_pixel, in_pixel, 0 ), scale_info->scale, user_data );
15098+ float coeff = 0.0f;
15099+
15100+ if ( ( weight >= stbir__too_small_coefficient ) || ( weight <= -stbir__too_small_coefficient ) )
15101+ coeff = (float) ( weight * scale_info->scale );
15102+
15103+ coefficient_group[i] = coeff;
15104+ }
15105+}
15106+
15107+// Normalize each output's coefficients: summed and scaled in stbir_real, each then rounded to float
15108+static void stbir__normalize_scatter_coefficients( stbir__contributors* contributors, float* coefficients, int coefficient_width, int num_contributors, int output_size, stbir_real too_small_sum )
15109+{
15110+ int i, j, first = 0;
15111+
15112+ for (i = 0; i < output_size; i++)
15113+ {
15114+ stbir_real total = 0;
15115+ stbir_real filter_scale;
15116+
15117+ while ( ( first < num_contributors ) && ( contributors[first].n1 < i ) )
15118+ first++;
15119+
15120+ for (j = first; j < num_contributors; j++)
15121+ {
15122+ if (i >= contributors[j].n0 && i <= contributors[j].n1)
15123+ total += (stbir_real) coefficients[ (size_t)j * coefficient_width + i - contributors[j].n0 ];
15124+ else if (i < contributors[j].n0)
15125+ break;
15126+ }
15127+
15128+ if ( ( total < too_small_sum ) && ( total > -too_small_sum ) )
15129+ filter_scale = 0; // all coeffs are extremely small, just zero them
15130+ else if ( ( total < ( (stbir_real)1.0 - stbir__renorm_tolerance ) ) || ( total > ( (stbir_real)1.0 + stbir__renorm_tolerance ) ) )
15131+ filter_scale = (stbir_real)1.0 / total;
15132+ else
15133+ continue;
15134+
15135+ for (j = first; j < num_contributors; j++)
15136+ {
15137+ if (i >= contributors[j].n0 && i <= contributors[j].n1)
15138+ coefficients[ (size_t)j * coefficient_width + i - contributors[j].n0 ] = (float) ( coefficients[ (size_t)j * coefficient_width + i - contributors[j].n0 ] * filter_scale );
15139+ else if (i < contributors[j].n0)
15140+ break;
15141+ }
15142+ }
15143+}
15144+
15145+// Polyphase: only the first numerator outputs were calculated. From first_copy on, each contributor is the one a
15146+// denominator before it with its outputs moved along by the numerator (calculated ones keep their outputs, the copies follow).
15147+static void stbir__replicate_scatter_coefficients( stbir__contributors* contributors, float* coefficients, int coefficient_width, int num_contributors, int first_copy, int calculated_end, int numerator, int denominator, int output_size )
15148+{
15149+ int j, i;
15150+
15151+ for (j = first_copy; j < num_contributors; j++)
15152+ {
15153+ stbir__contributors* contributor = contributors + j;
15154+ stbir__contributors* source = contributors + j - denominator;
15155+ float* coefficient_group = coefficients + (size_t)j * coefficient_width;
15156+ float* source_group = coefficients + (size_t)( j - denominator ) * coefficient_width;
15157+
15158+ if ( j >= calculated_end )
15159+ {
15160+ contributor->n0 = 0;
15161+ contributor->n1 = -1;
15162+
15163+ // the whole source fits: copy it
15164+ if ( source->n1 + numerator < output_size )
15165+ {
15166+ contributor->n0 = source->n0 + numerator;
15167+ contributor->n1 = source->n1 + numerator;
15168+ STBIR_MEMCPY( coefficient_group, source_group, (size_t)( source->n1 - source->n0 + 1 ) * sizeof(float) );
15169+ continue;
15170+ }
15171+ }
15172+
15173+ // append the source's outputs that are in range
15174+ for (i = 0; i <= source->n1 - source->n0; i++)
15175+ {
15176+ int out = source->n0 + i + numerator;
15177+ if ( out >= output_size )
15178+ break;
15179+ if ( contributor->n1 < contributor->n0 )
15180+ {
15181+ contributor->n0 = contributor->n1 = out;
15182+ coefficient_group[0] = source_group[i];
15183+ }
15184+ else
15185+ stbir__insert_coeff( contributor, coefficient_group, out, source_group[i], coefficient_width );
15186+ }
15187+ }
15188+}
15189+
15190+// The out of range input scanlines: clamp and reflect add their weights to the scanlines they map to (right edge
15191+// ascending, then left edge descending - the gather's order), zero drops them, wrap keeps them.
15192+static void stbir__fold_scatter_edges( stbir__sampler * samp, int coefficient_width )
15193+{
15194+ stbir_edge edge = samp->edge;
15195+ int input_size = samp->scale_info.input_full_size;
15196+ int filter_pixel_margin = samp->filter_pixel_margin;
15197+ stbir__contributors* contributors = samp->contributors;
15198+ float* coefficients = samp->coefficients;
15199+ int n, i;
15200+
15201+ if ( ( edge == STBIR_EDGE_CLAMP ) || ( edge == STBIR_EDGE_REFLECT ) )
15202+ {
15203+ int pass;
15204+ for (pass = 0; pass < 2; pass++)
15205+ {
15206+ int from = pass ? -1 : input_size;
15207+ int to = pass ? -filter_pixel_margin - 1 : input_size + filter_pixel_margin;
15208+ int step = pass ? -1 : 1;
15209+ for (n = from; n != to; n += step)
15210+ {
15211+ stbir__contributors* contributor = contributors + n + filter_pixel_margin;
15212+ float* coefficient_group = coefficients + (size_t)( n + filter_pixel_margin ) * coefficient_width;
15213+ int folded = stbir__edge_wrap_slow[edge]( n, input_size ) + filter_pixel_margin;
15214+
15215+ for (i = 0; i <= contributor->n1 - contributor->n0; i++)
15216+ stbir__insert_coeff( contributors + folded, coefficients + (size_t)folded * coefficient_width, contributor->n0 + i, coefficient_group[i], coefficient_width );
15217+
15218+ contributor->n0 = 0;
15219+ contributor->n1 = -1;
15220+ }
15221+ }
15222+ }
15223+ else if ( edge == STBIR_EDGE_ZERO )
15224+ {
15225+ for (n = 0; n < filter_pixel_margin; n++)
15226+ {
15227+ contributors[n].n0 = 0;
15228+ contributors[n].n1 = -1;
15229+ contributors[input_size + 2 * filter_pixel_margin - 1 - n].n0 = 0;
15230+ contributors[input_size + 2 * filter_pixel_margin - 1 - n].n1 = -1;
15231+ }
15232+ }
15233+}
15234+
15235+// Optimize step - skip zero coefficients at either end, and pack the groups from the build width down to the final width
15236+// (an output past the width is dropped). Also notes the first and last input scanlines with any weight.
15237+static void stbir__pack_scatter_coefficients( stbir__sampler * samp, int build_width )
15238+{
15239+ int coefficient_width = samp->coefficient_width;
15240+ int filter_pixel_margin = samp->filter_pixel_margin;
15241+ int num_contributors = samp->num_contributors;
15242+ stbir__contributors* contributors = samp->contributors;
15243+ float* coefficients = samp->coefficients;
15244+ int j, i;
15245+
15246+ samp->extent_info.lowest = 0x7fffffff;
15247+ samp->extent_info.highest = -0x7fffffff;
15248+ samp->extent_info.widest = coefficient_width;
15249+
15250+ for (j = 0; j < num_contributors; j++)
15251+ {
15252+ stbir__contributors* contributor = contributors + j;
15253+ float const* source_group = coefficients + (size_t)j * build_width;
15254+ float* coefficient_group = coefficients + (size_t)j * coefficient_width; // at or before the source, so packing forward in place is safe
15255+ int range = contributor->n1 - contributor->n0 + 1;
15256+ int skip = 0;
15257+
15258+ while ( ( skip < range ) && ( source_group[skip] == 0.0f ) )
15259+ skip++;
15260+ range -= skip;
15261+ if ( range > coefficient_width )
15262+ range = coefficient_width;
15263+ while ( ( range > 0 ) && ( source_group[skip + range - 1] == 0.0f ) )
15264+ range--;
15265+
15266+ for (i = 0; i < range; i++)
15267+ coefficient_group[i] = source_group[i + skip];
15268+
15269+ if ( range <= 0 )
15270+ {
15271+ contributor->n0 = 0;
15272+ contributor->n1 = -1;
15273+ continue;
15274+ }
15275+ contributor->n0 += skip;
15276+ contributor->n1 = contributor->n0 + range - 1;
15277+
15278+ if ( j - filter_pixel_margin < samp->extent_info.lowest )
15279+ samp->extent_info.lowest = j - filter_pixel_margin;
15280+ samp->extent_info.highest = j - filter_pixel_margin;
15281+ }
15282+}
15283+
15284+static void stbir__calculate_scatter_filters( stbir__sampler * samp, stbir_real in_pixels_radius, stbir_real support, void * user_data )
15285+{
15286+ stbir__scale_info const * scale_info = &samp->scale_info;
15287+ int output_size = scale_info->output_sub_size;
15288+ int filter_pixel_margin = samp->filter_pixel_margin;
15289+ int num_contributors = samp->num_contributors;
15290+ int build_width = samp->coefficient_width + 1;
15291+ int polyphase = ( scale_info->scale_is_rational ) && ( (int)scale_info->scale_numerator < output_size );
15292+ int numerator = polyphase ? (int)scale_info->scale_numerator : output_size;
15293+ int calculated_end = num_contributors; // the first contributor with nothing calculated
15294+ int first_weighted = -1; // the first contributor with any weight
15295+ int n;
15296+
15297+ // Looping through in pixels
15298+ for (n = 0; n < num_contributors; n++)
15299+ {
15300+ int out_first_pixel, out_last_pixel;
15301+
15302+ stbir__input_out_pixel_range( scale_info, n - filter_pixel_margin, in_pixels_radius, support, &out_first_pixel, &out_last_pixel );
15303+
15304+ // when polyphase, you only have to do coeffs up to the numerator count
15305+ if ( out_first_pixel >= numerator )
15306+ {
15307+ calculated_end = n;
15308+ break;
15309+ }
15310+ if ( out_last_pixel >= numerator )
15311+ out_last_pixel = numerator - 1;
15312+
15313+ stbir__calculate_coefficients_scatter( samp->filter_kernel_new, samp->filter_kernel_old, scale_info, n - filter_pixel_margin, out_first_pixel, out_last_pixel, samp->contributors + n, samp->coefficients + (size_t)n * build_width, user_data );
15314+
15315+ if ( ( first_weighted < 0 ) && ( out_last_pixel >= out_first_pixel ) )
15316+ first_weighted = n;
15317+ }
15318+ for (n = calculated_end; n < num_contributors; n++)
15319+ {
15320+ samp->contributors[n].n0 = 0;
15321+ samp->contributors[n].n1 = -1;
15322+ }
15323+
15324+ stbir__normalize_scatter_coefficients( samp->contributors, samp->coefficients, build_width, calculated_end, numerator, stbir__too_small_coefficient_sum_for( scale_info ) );
15325+
15326+ // the period is the rational's, which can be a little off the true one, so the copies start where it says
15327+ if ( ( polyphase ) && ( first_weighted >= 0 ) )
15328+ stbir__replicate_scatter_coefficients( samp->contributors, samp->coefficients, build_width, num_contributors, first_weighted + (int)scale_info->scale_denominator, calculated_end, numerator, (int)scale_info->scale_denominator, output_size );
15329+
15330+ stbir__fold_scatter_edges( samp, build_width );
15331+ stbir__pack_scatter_coefficients( samp, build_width );
15332+}
15333+
15334+static void stbir__calculate_filters( stbir__sampler * samp, void * user_data STBIR_ONLY_PROFILE_BUILD_GET_INFO )
15335+{
15336+ stbir_real scale = samp->scale_info.scale;
15337+ stbir__kernel_callback_new * kernel_new = samp->filter_kernel_new;
15338+ stbir__kernel_callback * kernel_old = samp->filter_kernel_old;
15339+ stbir_real in_pixels_radius = stbir__filter_radius_in_input_pixels( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, &samp->scale_info, user_data );
15340+ stbir_real inv_scale = samp->scale_info.inv_scale;
15341+ int input_full_size = samp->scale_info.input_full_size;
15342+ int gather_num_contributors = samp->num_contributors;
15343+ stbir__contributors* gather_contributors = samp->contributors;
15344+ float * gather_coeffs = samp->coefficients;
15345+ int gather_coefficient_width = samp->coefficient_width;
15346+
15347+ // with shared polyphase sets the generated phases go after the prologue sets
15348+ if ( samp->shared_period )
15349+ gather_coeffs += (size_t)samp->shared_prologue * gather_coefficient_width;
15350+
15351+ switch ( samp->is_gather )
15352+ {
15353+ case 1: // gather upsample
15354+ {
15355+ stbir_real filter_support = in_pixels_radius; // the filter's own pixels are input pixels
15356+ stbir_real out_pixels_radius = filter_support * scale;
15357+
15358+ stbir__calculate_coefficients_for_gather_upsample( out_pixels_radius, filter_support, kernel_new, kernel_old, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width, samp->edge, user_data );
15359+
15360+ STBIR_PROFILE_BUILD_START( cleanup );
15361+ if ( samp->shared_period )
15362+ stbir__cleanup_shared_coefficients( samp );
15363+ else
15364+ stbir__cleanup_gathered_coefficients( samp->edge, &samp->extent_info, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width );
15365+ STBIR_PROFILE_BUILD_END( cleanup );
15366+ }
15367+ break;
15368+
15369+ case 0: // scatter downsample (only on vertical)
15370+ case 2: // gather downsample
15371+ {
15372+ stbir_real filter_support = in_pixels_radius / inv_scale; // in output pixels
15373+ int filter_pixel_margin = samp->filter_pixel_margin;
15374+ int input_end = input_full_size + filter_pixel_margin;
15375+
15376+ if ( !samp->is_gather )
15377+ {
15378+ STBIR_PROFILE_BUILD_START( scatter );
15379+ stbir__calculate_scatter_filters( samp, in_pixels_radius, filter_support, user_data );
15380+ STBIR_PROFILE_BUILD_END( scatter );
15381+ break;
15382+ }
15383+
15384+ stbir__calculate_coefficients_for_gather_downsample( -filter_pixel_margin, input_end, in_pixels_radius, filter_support, kernel_new, kernel_old, &samp->scale_info, gather_coefficient_width, gather_num_contributors, gather_contributors, gather_coeffs, user_data );
15385+
15386+ STBIR_PROFILE_BUILD_START( cleanup );
15387+ if ( samp->shared_period )
15388+ stbir__cleanup_shared_coefficients( samp );
15389+ else
15390+ stbir__cleanup_gathered_coefficients( samp->edge, &samp->extent_info, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width );
15391+ STBIR_PROFILE_BUILD_END( cleanup );
15392+ }
15393+ break;
15394+ }
15395+}
15396+
15397+
15398+
15399+//========================================================================================================
15400+// scanline decoders and encoders
15401+
15402+#define stbir__coder_min_num 1
15403+#define STB_IMAGE_RESIZE_DO_CODERS
15404+#include STBIR__HEADER_FILENAME
15405+
15406+#define stbir__decode_suffix BGRA
15407+#define stbir__decode_swizzle
15408+#define stbir__decode_order0 2
15409+#define stbir__decode_order1 1
15410+#define stbir__decode_order2 0
15411+#define stbir__decode_order3 3
15412+#define stbir__encode_order0 2
15413+#define stbir__encode_order1 1
15414+#define stbir__encode_order2 0
15415+#define stbir__encode_order3 3
15416+#define stbir__coder_min_num 4
15417+#define STB_IMAGE_RESIZE_DO_CODERS
15418+#include STBIR__HEADER_FILENAME
15419+
15420+#define stbir__decode_suffix ARGB
15421+#define stbir__decode_swizzle
15422+#define stbir__decode_order0 1
15423+#define stbir__decode_order1 2
15424+#define stbir__decode_order2 3
15425+#define stbir__decode_order3 0
15426+#define stbir__encode_order0 3
15427+#define stbir__encode_order1 0
15428+#define stbir__encode_order2 1
15429+#define stbir__encode_order3 2
15430+#define stbir__coder_min_num 4
15431+#define STB_IMAGE_RESIZE_DO_CODERS
15432+#include STBIR__HEADER_FILENAME
15433+
15434+#define stbir__decode_suffix ABGR
15435+#define stbir__decode_swizzle
15436+#define stbir__decode_order0 3
15437+#define stbir__decode_order1 2
15438+#define stbir__decode_order2 1
15439+#define stbir__decode_order3 0
15440+#define stbir__encode_order0 3
15441+#define stbir__encode_order1 2
15442+#define stbir__encode_order2 1
15443+#define stbir__encode_order3 0
15444+#define stbir__coder_min_num 4
15445+#define STB_IMAGE_RESIZE_DO_CODERS
15446+#include STBIR__HEADER_FILENAME
15447+
15448+#define stbir__decode_suffix AR
15449+#define stbir__decode_swizzle
15450+#define stbir__decode_order0 1
15451+#define stbir__decode_order1 0
15452+#define stbir__decode_order2 3
15453+#define stbir__decode_order3 2
15454+#define stbir__encode_order0 1
15455+#define stbir__encode_order1 0
15456+#define stbir__encode_order2 3
15457+#define stbir__encode_order3 2
15458+#define stbir__coder_min_num 2
15459+#define STB_IMAGE_RESIZE_DO_CODERS
15460+#include STBIR__HEADER_FILENAME
15461+
15462+
15463+// fancy alpha means we expand to keep both premultipied and non-premultiplied color channels
15464+static void stbir__fancy_alpha_weight_4ch( float * out_buffer, ptrdiff_t width_times_channels )
15465+{
15466+ float STBIR_STREAMOUT_PTR(*) out = out_buffer;
15467+ float const * end_decode = out_buffer + ( width_times_channels / 4 ) * 7; // decode buffer aligned to end of out_buffer
15468+ float STBIR_STREAMOUT_PTR(*) decode = (float*)end_decode - width_times_channels;
15469+
15470+ // fancy alpha is stored internally as R G B A Rpm Gpm Bpm
15471+
15472+ #ifdef STBIR_SIMD
15473+
15474+ #ifdef STBIR_SIMD8
15475+ decode += 16;
15476+ STBIR_NO_UNROLL_LOOP_START
15477+ while ( decode <= end_decode )
15478+ {
15479+ stbir__simdf8 d0,d1,a0,a1,p0,p1;
15480+ STBIR_NO_UNROLL(decode);
15481+ stbir__simdf8_load( d0, decode-16 );
15482+ stbir__simdf8_load( d1, decode-16+8 );
15483+ stbir__simdf8_0123to33333333( a0, d0 );
15484+ stbir__simdf8_0123to33333333( a1, d1 );
15485+ stbir__simdf8_mult( p0, a0, d0 );
15486+ stbir__simdf8_mult( p1, a1, d1 );
15487+ stbir__simdf8_bot4s( a0, d0, p0 );
15488+ stbir__simdf8_bot4s( a1, d1, p1 );
15489+ stbir__simdf8_top4s( d0, d0, p0 );
15490+ stbir__simdf8_top4s( d1, d1, p1 );
15491+ stbir__simdf8_store ( out, a0 );
15492+ stbir__simdf8_store ( out+7, d0 );
15493+ stbir__simdf8_store ( out+14, a1 );
15494+ stbir__simdf8_store ( out+21, d1 );
15495+ decode += 16;
15496+ out += 28;
15497+ }
15498+ decode -= 16;
15499+ #else
15500+ decode += 8;
15501+ STBIR_NO_UNROLL_LOOP_START
15502+ while ( decode <= end_decode )
15503+ {
15504+ stbir__simdf d0,a0,d1,a1,p0,p1;
15505+ STBIR_NO_UNROLL(decode);
15506+ stbir__simdf_load( d0, decode-8 );
15507+ stbir__simdf_load( d1, decode-8+4 );
15508+ stbir__simdf_0123to3333( a0, d0 );
15509+ stbir__simdf_0123to3333( a1, d1 );
15510+ stbir__simdf_mult( p0, a0, d0 );
15511+ stbir__simdf_mult( p1, a1, d1 );
15512+ stbir__simdf_store ( out, d0 );
15513+ stbir__simdf_store ( out+4, p0 );
15514+ stbir__simdf_store ( out+7, d1 );
15515+ stbir__simdf_store ( out+7+4, p1 );
15516+ decode += 8;
15517+ out += 14;
15518+ }
15519+ decode -= 8;
15520+ #endif
15521+
15522+ // might be one last odd pixel
15523+ #ifdef STBIR_SIMD8
15524+ STBIR_NO_UNROLL_LOOP_START
15525+ while ( decode < end_decode )
15526+ #else
15527+ if ( decode < end_decode )
15528+ #endif
15529+ {
15530+ stbir__simdf d,a,p;
15531+ STBIR_NO_UNROLL(decode);
15532+ stbir__simdf_load( d, decode );
15533+ stbir__simdf_0123to3333( a, d );
15534+ stbir__simdf_mult( p, a, d );
15535+ stbir__simdf_store ( out, d );
15536+ stbir__simdf_store ( out+4, p );
15537+ decode += 4;
15538+ out += 7;
15539+ }
15540+
15541+ #else
15542+
15543+ while( decode < end_decode )
15544+ {
15545+ float r = decode[0], g = decode[1], b = decode[2], alpha = decode[3];
15546+ out[0] = r;
15547+ out[1] = g;
15548+ out[2] = b;
15549+ out[3] = alpha;
15550+ out[4] = r * alpha;
15551+ out[5] = g * alpha;
15552+ out[6] = b * alpha;
15553+ out += 7;
15554+ decode += 4;
15555+ }
15556+
15557+ #endif
15558+}
15559+
15560+static void stbir__fancy_alpha_weight_2ch( float * out_buffer, ptrdiff_t width_times_channels )
15561+{
15562+ float STBIR_STREAMOUT_PTR(*) out = out_buffer;
15563+ float const * end_decode = out_buffer + ( width_times_channels / 2 ) * 3;
15564+ float STBIR_STREAMOUT_PTR(*) decode = (float*)end_decode - width_times_channels;
15565+
15566+ // for fancy alpha, turns into: [X A Xpm][X A Xpm],etc
15567+
15568+ #ifdef STBIR_SIMD
15569+
15570+ decode += 8;
15571+ if ( decode <= end_decode )
15572+ {
15573+ STBIR_NO_UNROLL_LOOP_START
15574+ do {
15575+ #ifdef STBIR_SIMD8
15576+ stbir__simdf8 d0,a0,p0;
15577+ STBIR_NO_UNROLL(decode);
15578+ stbir__simdf8_load( d0, decode-8 );
15579+ stbir__simdf8_0123to11331133( p0, d0 );
15580+ stbir__simdf8_0123to00220022( a0, d0 );
15581+ stbir__simdf8_mult( p0, p0, a0 );
15582+
15583+ stbir__simdf_store2( out, stbir__if_simdf8_cast_to_simdf4( d0 ) );
15584+ stbir__simdf_store( out+2, stbir__if_simdf8_cast_to_simdf4( p0 ) );
15585+ stbir__simdf_store2h( out+3, stbir__if_simdf8_cast_to_simdf4( d0 ) );
15586+
15587+ stbir__simdf_store2( out+6, stbir__simdf8_gettop4( d0 ) );
15588+ stbir__simdf_store( out+8, stbir__simdf8_gettop4( p0 ) );
15589+ stbir__simdf_store2h( out+9, stbir__simdf8_gettop4( d0 ) );
15590+ #else
15591+ stbir__simdf d0,a0,d1,a1,p0,p1;
15592+ STBIR_NO_UNROLL(decode);
15593+ stbir__simdf_load( d0, decode-8 );
15594+ stbir__simdf_load( d1, decode-8+4 );
15595+ stbir__simdf_0123to1133( p0, d0 );
15596+ stbir__simdf_0123to1133( p1, d1 );
15597+ stbir__simdf_0123to0022( a0, d0 );
15598+ stbir__simdf_0123to0022( a1, d1 );
15599+ stbir__simdf_mult( p0, p0, a0 );
15600+ stbir__simdf_mult( p1, p1, a1 );
15601+
15602+ stbir__simdf_store2( out, d0 );
15603+ stbir__simdf_store( out+2, p0 );
15604+ stbir__simdf_store2h( out+3, d0 );
15605+
15606+ stbir__simdf_store2( out+6, d1 );
15607+ stbir__simdf_store( out+8, p1 );
15608+ stbir__simdf_store2h( out+9, d1 );
15609+ #endif
15610+ decode += 8;
15611+ out += 12;
15612+ } while ( decode <= end_decode );
15613+ }
15614+ decode -= 8;
15615+ #endif
15616+
15617+ STBIR_SIMD_NO_UNROLL_LOOP_START
15618+ while( decode < end_decode )
15619+ {
15620+ float x = decode[0], y = decode[1];
15621+ STBIR_SIMD_NO_UNROLL(decode);
15622+ out[0] = x;
15623+ out[1] = y;
15624+ out[2] = x * y;
15625+ out += 3;
15626+ decode += 2;
15627+ }
15628+}
15629+
15630+static void stbir__fancy_alpha_unweight_4ch( float * encode_buffer, ptrdiff_t width_times_channels )
15631+{
15632+ float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
15633+ float STBIR_SIMD_STREAMOUT_PTR(*) input = encode_buffer;
15634+ float const * end_output = encode_buffer + width_times_channels;
15635+
15636+ // fancy RGBA is stored internally as R G B A Rpm Gpm Bpm
15637+
15638+ STBIR_SIMD_NO_UNROLL_LOOP_START
15639+ do {
15640+ float alpha = input[3];
15641+#ifdef STBIR_SIMD
15642+ stbir__simdf i,ia;
15643+ STBIR_SIMD_NO_UNROLL(encode);
15644+ if ( alpha < stbir__alpha_too_small_to_divide )
15645+ {
15646+ stbir__simdf_load( i, input );
15647+ stbir__simdf_store( encode, i );
15648+ }
15649+ else
15650+ {
15651+ stbir__simdf_load1frep4( ia, 1.0f / alpha );
15652+ stbir__simdf_load( i, input+4 );
15653+ stbir__simdf_mult( i, i, ia );
15654+ stbir__simdf_store( encode, i );
15655+ encode[3] = alpha;
15656+ }
15657+#else
15658+ if ( alpha < stbir__alpha_too_small_to_divide )
15659+ {
15660+ encode[0] = input[0];
15661+ encode[1] = input[1];
15662+ encode[2] = input[2];
15663+ }
15664+ else
15665+ {
15666+ float ialpha = 1.0f / alpha;
15667+ encode[0] = input[4] * ialpha;
15668+ encode[1] = input[5] * ialpha;
15669+ encode[2] = input[6] * ialpha;
15670+ }
15671+ encode[3] = alpha;
15672+#endif
15673+
15674+ input += 7;
15675+ encode += 4;
15676+ } while ( encode < end_output );
15677+}
15678+
15679+// format: [X A Xpm][X A Xpm] etc
15680+static void stbir__fancy_alpha_unweight_2ch( float * encode_buffer, ptrdiff_t width_times_channels )
15681+{
15682+ float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
15683+ float STBIR_SIMD_STREAMOUT_PTR(*) input = encode_buffer;
15684+ float const * end_output = encode_buffer + width_times_channels;
15685+
15686+ do {
15687+ float alpha = input[1];
15688+ encode[0] = input[0];
15689+ if ( alpha >= stbir__alpha_too_small_to_divide )
15690+ encode[0] = input[2] / alpha;
15691+ encode[1] = alpha;
15692+
15693+ input += 3;
15694+ encode += 2;
15695+ } while ( encode < end_output );
15696+}
15697+
15698+static void stbir__simple_alpha_weight_4ch( float * decode_buffer, ptrdiff_t width_times_channels )
15699+{
15700+ float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
15701+ float const * end_decode = decode_buffer + width_times_channels;
15702+
15703+ #ifdef STBIR_SIMD
15704+ {
15705+ decode += 2 * stbir__simdfX_float_count;
15706+ STBIR_NO_UNROLL_LOOP_START
15707+ while ( decode <= end_decode )
15708+ {
15709+ stbir__simdfX d0,a0,d1,a1;
15710+ STBIR_NO_UNROLL(decode);
15711+ stbir__simdfX_load( d0, decode-2*stbir__simdfX_float_count );
15712+ stbir__simdfX_load( d1, decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count );
15713+ stbir__simdfX_aaa1( a0, d0, STBIR_onesX );
15714+ stbir__simdfX_aaa1( a1, d1, STBIR_onesX );
15715+ stbir__simdfX_mult( d0, d0, a0 );
15716+ stbir__simdfX_mult( d1, d1, a1 );
15717+ stbir__simdfX_store ( decode-2*stbir__simdfX_float_count, d0 );
15718+ stbir__simdfX_store ( decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count, d1 );
15719+ decode += 2 * stbir__simdfX_float_count;
15720+ }
15721+ decode -= 2 * stbir__simdfX_float_count;
15722+
15723+ // few last pixels remnants
15724+ #ifdef STBIR_SIMD8
15725+ STBIR_NO_UNROLL_LOOP_START
15726+ while ( decode < end_decode )
15727+ #else
15728+ if ( decode < end_decode )
15729+ #endif
15730+ {
15731+ stbir__simdf d,a;
15732+ stbir__simdf_load( d, decode );
15733+ stbir__simdf_aaa1( a, d, STBIR__CONSTF(STBIR_ones) );
15734+ stbir__simdf_mult( d, d, a );
15735+ stbir__simdf_store ( decode, d );
15736+ decode += 4;
15737+ }
15738+ }
15739+
15740+ #else
15741+
15742+ while( decode < end_decode )
15743+ {
15744+ float alpha = decode[3];
15745+ decode[0] *= alpha;
15746+ decode[1] *= alpha;
15747+ decode[2] *= alpha;
15748+ decode += 4;
15749+ }
15750+
15751+ #endif
15752+}
15753+
15754+static void stbir__simple_alpha_weight_2ch( float * decode_buffer, ptrdiff_t width_times_channels )
15755+{
15756+ float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
15757+ float const * end_decode = decode_buffer + width_times_channels;
15758+
15759+ #ifdef STBIR_SIMD
15760+ decode += 2 * stbir__simdfX_float_count;
15761+ STBIR_NO_UNROLL_LOOP_START
15762+ while ( decode <= end_decode )
15763+ {
15764+ stbir__simdfX d0,a0,d1,a1;
15765+ STBIR_NO_UNROLL(decode);
15766+ stbir__simdfX_load( d0, decode-2*stbir__simdfX_float_count );
15767+ stbir__simdfX_load( d1, decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count );
15768+ stbir__simdfX_a1a1( a0, d0, STBIR_onesX );
15769+ stbir__simdfX_a1a1( a1, d1, STBIR_onesX );
15770+ stbir__simdfX_mult( d0, d0, a0 );
15771+ stbir__simdfX_mult( d1, d1, a1 );
15772+ stbir__simdfX_store ( decode-2*stbir__simdfX_float_count, d0 );
15773+ stbir__simdfX_store ( decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count, d1 );
15774+ decode += 2 * stbir__simdfX_float_count;
15775+ }
15776+ decode -= 2 * stbir__simdfX_float_count;
15777+ #endif
15778+
15779+ STBIR_SIMD_NO_UNROLL_LOOP_START
15780+ while( decode < end_decode )
15781+ {
15782+ float alpha = decode[1];
15783+ STBIR_SIMD_NO_UNROLL(decode);
15784+ decode[0] *= alpha;
15785+ decode += 2;
15786+ }
15787+}
15788+
15789+static void stbir__simple_alpha_unweight_4ch( float * encode_buffer, ptrdiff_t width_times_channels )
15790+{
15791+ float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
15792+ float const * end_output = encode_buffer + width_times_channels;
15793+
15794+ STBIR_SIMD_NO_UNROLL_LOOP_START
15795+ do {
15796+ float alpha = encode[3];
15797+
15798+#ifdef STBIR_SIMD
15799+ stbir__simdf i,ia;
15800+ STBIR_SIMD_NO_UNROLL(encode);
15801+ if ( alpha >= stbir__alpha_too_small_to_divide )
15802+ {
15803+ stbir__simdf_load1frep4( ia, 1.0f / alpha );
15804+ stbir__simdf_load( i, encode );
15805+ stbir__simdf_mult( i, i, ia );
15806+ stbir__simdf_store( encode, i );
15807+ encode[3] = alpha;
15808+ }
15809+#else
15810+ if ( alpha >= stbir__alpha_too_small_to_divide )
15811+ {
15812+ float ialpha = 1.0f / alpha;
15813+ encode[0] *= ialpha;
15814+ encode[1] *= ialpha;
15815+ encode[2] *= ialpha;
15816+ }
15817+#endif
15818+ encode += 4;
15819+ } while ( encode < end_output );
15820+}
15821+
15822+static void stbir__simple_alpha_unweight_2ch( float * encode_buffer, ptrdiff_t width_times_channels )
15823+{
15824+ float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
15825+ float const * end_output = encode_buffer + width_times_channels;
15826+
15827+ do {
15828+ float alpha = encode[1];
15829+ if ( alpha >= stbir__alpha_too_small_to_divide )
15830+ encode[0] /= alpha;
15831+ encode += 2;
15832+ } while ( encode < end_output );
15833+}
15834+
15835+
15836+// only used in RGB->BGR or BGR->RGB
15837+static void stbir__simple_flip_3ch( float * decode_buffer, ptrdiff_t width_times_channels )
15838+{
15839+ float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
15840+ float const * end_decode = decode_buffer + width_times_channels;
15841+
15842+#ifdef STBIR_SIMD
15843+ #ifdef stbir__simdf_swiz2 // do we have two argument swizzles?
15844+ end_decode -= 12;
15845+ STBIR_NO_UNROLL_LOOP_START
15846+ while( decode <= end_decode )
15847+ {
15848+ // on arm64 8 instructions, no overlapping stores
15849+ stbir__simdf a,b,c,na,nb;
15850+ STBIR_SIMD_NO_UNROLL(decode);
15851+ stbir__simdf_load( a, decode );
15852+ stbir__simdf_load( b, decode+4 );
15853+ stbir__simdf_load( c, decode+8 );
15854+
15855+ na = stbir__simdf_swiz2( a, b, 2, 1, 0, 5 );
15856+ b = stbir__simdf_swiz2( a, b, 4, 3, 6, 7 );
15857+ nb = stbir__simdf_swiz2( b, c, 0, 1, 4, 3 );
15858+ c = stbir__simdf_swiz2( b, c, 2, 7, 6, 5 );
15859+
15860+ stbir__simdf_store( decode, na );
15861+ stbir__simdf_store( decode+4, nb );
15862+ stbir__simdf_store( decode+8, c );
15863+ decode += 12;
15864+ }
15865+ end_decode += 12;
15866+ #else
15867+ end_decode -= 24;
15868+ STBIR_NO_UNROLL_LOOP_START
15869+ while( decode <= end_decode )
15870+ {
15871+ // 26 instructions on x64
15872+ stbir__simdf a,b,c,d,e,f,g;
15873+ float i21, i23;
15874+ STBIR_SIMD_NO_UNROLL(decode);
15875+ stbir__simdf_load( a, decode );
15876+ stbir__simdf_load( b, decode+3 );
15877+ stbir__simdf_load( c, decode+6 );
15878+ stbir__simdf_load( d, decode+9 );
15879+ stbir__simdf_load( e, decode+12 );
15880+ stbir__simdf_load( f, decode+15 );
15881+ stbir__simdf_load( g, decode+18 );
15882+
15883+ a = stbir__simdf_swiz( a, 2, 1, 0, 3 );
15884+ b = stbir__simdf_swiz( b, 2, 1, 0, 3 );
15885+ c = stbir__simdf_swiz( c, 2, 1, 0, 3 );
15886+ d = stbir__simdf_swiz( d, 2, 1, 0, 3 );
15887+ e = stbir__simdf_swiz( e, 2, 1, 0, 3 );
15888+ f = stbir__simdf_swiz( f, 2, 1, 0, 3 );
15889+ g = stbir__simdf_swiz( g, 2, 1, 0, 3 );
15890+
15891+ // stores overlap, need to be in order,
15892+ stbir__simdf_store( decode, a );
15893+ i21 = decode[21];
15894+ stbir__simdf_store( decode+3, b );
15895+ i23 = decode[23];
15896+ stbir__simdf_store( decode+6, c );
15897+ stbir__simdf_store( decode+9, d );
15898+ stbir__simdf_store( decode+12, e );
15899+ stbir__simdf_store( decode+15, f );
15900+ stbir__simdf_store( decode+18, g );
15901+ decode[21] = i23;
15902+ decode[23] = i21;
15903+ decode += 24;
15904+ }
15905+ end_decode += 24;
15906+ #endif
15907+#else
15908+ end_decode -= 12;
15909+ STBIR_NO_UNROLL_LOOP_START
15910+ while( decode <= end_decode )
15911+ {
15912+ // 16 instructions
15913+ float t0,t1,t2,t3;
15914+ STBIR_NO_UNROLL(decode);
15915+ t0 = decode[0];
15916+ t1 = decode[3];
15917+ t2 = decode[6];
15918+ t3 = decode[9];
15919+ decode[0] = decode[2];
15920+ decode[3] = decode[5];
15921+ decode[6] = decode[8];
15922+ decode[9] = decode[11];
15923+ decode[2] = t0;
15924+ decode[5] = t1;
15925+ decode[8] = t2;
15926+ decode[11] = t3;
15927+ decode += 12;
15928+ }
15929+ end_decode += 12;
15930+#endif
15931+
15932+ STBIR_NO_UNROLL_LOOP_START
15933+ while( decode < end_decode )
15934+ {
15935+ float t = decode[0];
15936+ STBIR_NO_UNROLL(decode);
15937+ decode[0] = decode[2];
15938+ decode[2] = t;
15939+ decode += 3;
15940+ }
15941+}
15942+
15943+
15944+// decodes the input pixels that a column slice needs for input scanline n into output_buffer
15945+// most of the complexity in this function is for wrap mode, we need to grab from multiple
15946+// points in the input buffer
15947+static void stbir__decode_scanline(stbir__info const * stbir_info, int n, float * output_buffer, stbir__column_slice const * slice STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
15948+{
15949+ int channels = stbir_info->channels;
15950+ int effective_channels = stbir_info->effective_channels;
15951+ int input_sample_in_bytes = stbir__type_size[stbir_info->input_type] * channels;
15952+ stbir_edge edge_horizontal = stbir_info->horizontal.edge;
15953+ stbir_edge edge_vertical = stbir_info->vertical.edge;
15954+ int input_full_size = stbir_info->horizontal.scale_info.input_full_size;
15955+ int row = stbir__edge_wrap(edge_vertical, n, stbir_info->vertical.scale_info.input_full_size);
15956+ const void* input_plane_data = ( (char *) stbir_info->input_data ) + (ptrdiff_t)( row * stbir_info->input_stride_bytes );
15957+ float * full_decode_buffer = output_buffer - (ptrdiff_t)slice->in_n0 * effective_channels; // decode buffer is indexed in input pixel space, so back it up to input pixel zero (the products are 64-bit: a tile of a huge input can start past 2^31 / channels)
15958+ float * last_decoded = output_buffer;
15959+ int run_n0[3], run_n1[3], run_input_x[3];
15960+ int num_runs = 0, r;
15961+ int n0 = slice->in_n0, n1 = slice->in_n1;
15962+
15963+ // if we are on edge_zero, and we get in here with an out of bounds n, then the calculate filters has failed
15964+ STBIR_ASSERT( !(edge_vertical == STBIR_EDGE_ZERO && (n < 0 || n >= stbir_info->vertical.scale_info.input_full_size)) );
15965+
15966+ if ( edge_horizontal == STBIR_EDGE_WRAP )
15967+ {
15968+ // the margins are never wider than the scanline (stbir__set_sampler clamps them), so each side wraps exactly once
15969+ STBIR_ASSERT( n0 >= -input_full_size );
15970+ if ( n1 > ( ( input_full_size - 1 ) + input_full_size ) )
15971+ n1 = ( input_full_size - 1 ) + input_full_size; // the conservative range can poke one past what the filter can reach, that pixel just gets zeroed below ( 2 x 2^30 - 1, written so it can't overflow )
15972+
15973+ if ( n0 < 0 )
15974+ {
15975+ run_n0[num_runs] = n0;
15976+ run_n1[num_runs] = ( n1 < -1 ) ? n1 : -1;
15977+ run_input_x[num_runs] = n0 + input_full_size;
15978+ ++num_runs;
15979+ }
15980+ if ( ( n1 >= 0 ) && ( n0 < input_full_size ) )
15981+ {
15982+ run_n0[num_runs] = ( n0 > 0 ) ? n0 : 0;
15983+ run_n1[num_runs] = ( n1 < ( input_full_size - 1 ) ) ? n1 : ( input_full_size - 1 );
15984+ run_input_x[num_runs] = run_n0[num_runs];
15985+ ++num_runs;
15986+ }
15987+ if ( n1 >= input_full_size )
15988+ {
15989+ run_n0[num_runs] = ( n0 > input_full_size ) ? n0 : input_full_size;
15990+ run_n1[num_runs] = n1;
15991+ run_input_x[num_runs] = run_n0[num_runs] - input_full_size;
15992+ ++num_runs;
15993+ }
15994+ }
15995+ else
15996+ {
15997+ STBIR_ASSERT( ( n0 >= 0 ) && ( n1 < input_full_size ) );
15998+ run_n0[0] = n0;
15999+ run_n1[0] = n1;
16000+ run_input_x[0] = n0;
16001+ num_runs = 1;
16002+ }
16003+
16004+ for( r = 0 ; r < num_runs ; r++ )
16005+ {
16006+ int width = run_n1[r] + 1 - run_n0[r];
16007+ ptrdiff_t width_times_channels = (ptrdiff_t)width * channels;
16008+ float * decode_buffer = full_decode_buffer + (ptrdiff_t)run_n0[r] * effective_channels;
16009+ float * end_decode = full_decode_buffer + (ptrdiff_t)( run_n1[r] + 1 ) * effective_channels;
16010+ void const * input_data;
16011+
16012+ // read directly out of input plane by default
16013+ input_data = ( (char*)input_plane_data ) + (ptrdiff_t)run_input_x[r] * input_sample_in_bytes;
16014+
16015+ // if we have an input callback, call it to get the input data
16016+ if ( stbir_info->in_pixels_cb )
16017+ {
16018+ // call the callback with a temp buffer (that they can choose to use or not). the temp is just right aligned memory in the decode_buffer itself
16019+ input_data = stbir_info->in_pixels_cb( ( (char*) end_decode ) - ( (ptrdiff_t)width * input_sample_in_bytes ) + stbir_info->input_callback_offset_bytes, input_plane_data, width, run_input_x[r], row, stbir_info->user_data );
16020+ }
16021+
16022+ STBIR_PROFILE_START( decode );
16023+ // convert the pixels info the float decode_buffer, (we index from end_decode, so that when channels<effective_channels, we are right justified in the buffer)
16024+ last_decoded = stbir_info->decode_pixels( (float*)end_decode - width_times_channels, width_times_channels, input_data );
16025+ STBIR_PROFILE_END( decode );
16026+
16027+ if (stbir_info->alpha_weight)
16028+ {
16029+ STBIR_PROFILE_START( alpha );
16030+ stbir_info->alpha_weight( decode_buffer, width_times_channels );
16031+ STBIR_PROFILE_END( alpha );
16032+ }
16033+ }
16034+
16035+ // the horizontal kernels sometimes read a few samples past the input end, so
16036+ // zero the extra to avoid NaN (has to happen after decoding, since the input
16037+ // buffer might overwrite them. Should just be a couple clears.
16038+ {
16039+ float * range_end = full_decode_buffer + (ptrdiff_t)( slice->in_n1 + 1 ) * effective_channels + 2;
16040+ while ( last_decoded < range_end )
16041+ *last_decoded++ = 0.0f;
16042+ }
16043+}
16044+
16045+// advance to the next set of coeffs and contrib info
16046+#define stbir__next_contributor() \
16047+ decode_buffer += column->advance; \
16048+ column = ( (stbir__column_entry const *)( ( (char const *)(column) ) + (entry_stride) ) );
16049+
16050+
16051+//=================
16052+// Do 1 channel horizontal routines
16053+
16054+#ifdef STBIR_SIMD
16055+
16056+#define stbir__1_coeff_only() \
16057+ stbir__simdf tot,c; \
16058+ STBIR_SIMD_NO_UNROLL(decode); \
16059+ stbir__simdf_load1( c, hc ); \
16060+ stbir__simdf_mult1_mem( tot, c, decode );
16061+
16062+#define stbir__2_coeff_only() \
16063+ stbir__simdf tot,c,d; \
16064+ STBIR_SIMD_NO_UNROLL(decode); \
16065+ stbir__simdf_load2z( c, hc ); \
16066+ stbir__simdf_load2( d, decode ); \
16067+ stbir__simdf_mult( tot, c, d ); \
16068+ stbir__simdf_0123to1230( c, tot ); \
16069+ stbir__simdf_add1( tot, tot, c );
16070+
16071+#define stbir__3_coeff_only() \
16072+ stbir__simdf tot,c,t; \
16073+ STBIR_SIMD_NO_UNROLL(decode); \
16074+ stbir__simdf_load( c, hc ); \
16075+ stbir__simdf_mult_mem( tot, c, decode ); \
16076+ stbir__simdf_0123to1230( c, tot ); \
16077+ stbir__simdf_0123to2301( t, tot ); \
16078+ stbir__simdf_add1( tot, tot, c ); \
16079+ stbir__simdf_add1( tot, tot, t );
16080+
16081+#define stbir__store_output_tiny() \
16082+ stbir__simdf_store1( output, tot ); \
16083+ stbir__next_contributor(); \
16084+ output += 1;
16085+
16086+#define stbir__4_coeff_start() \
16087+ stbir__simdf tot,c; \
16088+ STBIR_SIMD_NO_UNROLL(decode); \
16089+ stbir__simdf_load( c, hc ); \
16090+ stbir__simdf_mult_mem( tot, c, decode ); \
16091+
16092+#define stbir__4_coeff_continue_from_4( ofs ) \
16093+ STBIR_SIMD_NO_UNROLL(decode); \
16094+ stbir__simdf_load( c, hc + (ofs) ); \
16095+ stbir__simdf_madd_mem( tot, tot, c, decode+(ofs) );
16096+
16097+#define stbir__1_coeff_remnant( ofs ) \
16098+ { stbir__simdf d; \
16099+ stbir__simdf_load1z( c, hc + (ofs) ); \
16100+ stbir__simdf_load1( d, decode + (ofs) ); \
16101+ stbir__simdf_madd( tot, tot, d, c ); }
16102+
16103+#define stbir__2_coeff_remnant( ofs ) \
16104+ { stbir__simdf d; \
16105+ stbir__simdf_load2z( c, hc+(ofs) ); \
16106+ stbir__simdf_load2( d, decode+(ofs) ); \
16107+ stbir__simdf_madd( tot, tot, d, c ); }
16108+
16109+#define stbir__3_coeff_setup() \
16110+ stbir__simdf mask; \
16111+ stbir__simdf_load( mask, STBIR_mask + 3 );
16112+
16113+#define stbir__3_coeff_remnant( ofs ) \
16114+ stbir__simdf_load( c, hc+(ofs) ); \
16115+ stbir__simdf_and( c, c, mask ); \
16116+ stbir__simdf_madd_mem( tot, tot, c, decode+(ofs) );
16117+
16118+// four outputs per iteration for one channel - do 4 1-channel versions
16119+#define stbir__x4_setup() \
16120+ stbir__column_entry const * e1 = (stbir__column_entry const *)( (char const *)column + entry_stride ); \
16121+ stbir__column_entry const * e2 = (stbir__column_entry const *)( (char const *)e1 + entry_stride ); \
16122+ stbir__column_entry const * e3 = (stbir__column_entry const *)( (char const *)e2 + entry_stride ); \
16123+ float const * d0 = decode_buffer, * d1 = d0 + column->advance, * d2 = d1 + e1->advance, * d3 = d2 + e2->advance; \
16124+ float const * h0 = column->coefficients, * h1 = e1->coefficients, * h2 = e2->coefficients, * h3 = e3->coefficients; \
16125+ stbir__simdf a0, a1, a2, a3, c;
16126+
16127+#define stbir__x4_start() \
16128+ do { \
16129+ stbir__simdf_load( c, h0 ); \
16130+ stbir__simdf_mult_mem( a0, c, d0 ); \
16131+ stbir__simdf_load( c, h1 ); \
16132+ stbir__simdf_mult_mem( a1, c, d1 ); \
16133+ stbir__simdf_load( c, h2 ); \
16134+ stbir__simdf_mult_mem( a2, c, d2 ); \
16135+ stbir__simdf_load( c, h3 ); \
16136+ stbir__simdf_mult_mem( a3, c, d3 ); \
16137+ } while(0)
16138+
16139+#define stbir__x4_cont( ofs ) \
16140+ do { \
16141+ stbir__simdf_load( c, h0 + (ofs) ); \
16142+ stbir__simdf_madd_mem( a0, a0, c, d0 + (ofs) ); \
16143+ stbir__simdf_load( c, h1 + (ofs) ); \
16144+ stbir__simdf_madd_mem( a1, a1, c, d1 + (ofs) ); \
16145+ stbir__simdf_load( c, h2 + (ofs) ); \
16146+ stbir__simdf_madd_mem( a2, a2, c, d2 + (ofs) ); \
16147+ stbir__simdf_load( c, h3 + (ofs) ); \
16148+ stbir__simdf_madd_mem( a3, a3, c, d3 + (ofs) ); \
16149+ } while(0)
16150+
16151+#define stbir__x4_rem1( ofs ) \
16152+ do { \
16153+ stbir__simdf d; \
16154+ stbir__simdf_load1z( c, h0 + (ofs) ); \
16155+ stbir__simdf_load1( d, d0 + (ofs) ); \
16156+ stbir__simdf_madd( a0, a0, d, c ); \
16157+ stbir__simdf_load1z( c, h1 + (ofs) ); \
16158+ stbir__simdf_load1( d, d1 + (ofs) ); \
16159+ stbir__simdf_madd( a1, a1, d, c ); \
16160+ stbir__simdf_load1z( c, h2 + (ofs) ); \
16161+ stbir__simdf_load1( d, d2 + (ofs) ); \
16162+ stbir__simdf_madd( a2, a2, d, c ); \
16163+ stbir__simdf_load1z( c, h3 + (ofs) ); \
16164+ stbir__simdf_load1( d, d3 + (ofs) ); \
16165+ stbir__simdf_madd( a3, a3, d, c ); \
16166+ } while(0)
16167+
16168+#define stbir__x4_rem2( ofs ) \
16169+ do { \
16170+ stbir__simdf d; \
16171+ stbir__simdf_load2z( c, h0 + (ofs) ); \
16172+ stbir__simdf_load2( d, d0 + (ofs) ); \
16173+ stbir__simdf_madd( a0, a0, d, c ); \
16174+ stbir__simdf_load2z( c, h1 + (ofs) ); \
16175+ stbir__simdf_load2( d, d1 + (ofs) ); \
16176+ stbir__simdf_madd( a1, a1, d, c ); \
16177+ stbir__simdf_load2z( c, h2 + (ofs) ); \
16178+ stbir__simdf_load2( d, d2 + (ofs) ); \
16179+ stbir__simdf_madd( a2, a2, d, c ); \
16180+ stbir__simdf_load2z( c, h3 + (ofs) ); \
16181+ stbir__simdf_load2( d, d3 + (ofs) ); \
16182+ stbir__simdf_madd( a3, a3, d, c ); \
16183+ } while(0)
16184+
16185+// widths of 4n+3 carry a zero after their last coefficient (STBIR__COEFFICIENT_PAD), so three is a full four
16186+#define stbir__x4_rem3( ofs ) stbir__x4_cont( ofs )
16187+
16188+// two coefficients: ( d0 d0+1 | d1 d1+1 ) * ( c0 c0+1 | c1 c1+1 ), twice, then one pairwise add gives the four outputs
16189+#define stbir__x4_2taps() \
16190+ do { \
16191+ stbir__simdf_load2( c, h0 ); \
16192+ stbir__simdf_load2hmerge( c, c, h1 ); \
16193+ stbir__simdf_load2( a0, d0 ); \
16194+ stbir__simdf_load2hmerge( a0, a0, d1 ); \
16195+ stbir__simdf_mult( a0, a0, c ); \
16196+ stbir__simdf_load2( a2, h2 ); \
16197+ stbir__simdf_load2hmerge( a2, a2, h3 ); \
16198+ stbir__simdf_load2( a3, d2 ); \
16199+ stbir__simdf_load2hmerge( a3, a3, d3 ); \
16200+ stbir__simdf_mult( a1, a3, a2 ); \
16201+ stbir__simdf_hadd2( a0, a0, a1 ); \
16202+ } while(0)
16203+
16204+// the same association as the per pixel path's reduction, ( l0 + l2 ) + ( l1 + l3 ), so the results match it bit for bit
16205+#define stbir__x4_finish() \
16206+ do { \
16207+ stbir__simdf_addhalves2( a0, a0, a1 ); \
16208+ stbir__simdf_addhalves2( a2, a2, a3 ); \
16209+ stbir__simdf_hadd2( a0, a0, a2 ); \
16210+ } while(0)
16211+
16212+// the three coefficient per pixel path reduces as ( l0 + l1 ) + l2, which the pairwise tree reproduces (lane 3 is the zero pad)
16213+#define stbir__x4_finish3() \
16214+ do { \
16215+ stbir__simdf_hadd2( a0, a0, a1 ); \
16216+ stbir__simdf_hadd2( a2, a2, a3 ); \
16217+ stbir__simdf_hadd2( a0, a0, a2 ); \
16218+ } while(0)
16219+
16220+#define stbir__x4_store() \
16221+ do { \
16222+ stbir__simdf_store( output, a0 ); \
16223+ decode_buffer = d3 + e3->advance; \
16224+ column = (stbir__column_entry const *)( (char const *)e3 + entry_stride ); \
16225+ output += 4; \
16226+ } while(0)
16227+
16228+// polyphase period 1 for 1 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
16229+#define stbir__period1_setup_4() stbir__simdf P0; stbir__simdf_load( P0, hc );
16230+#define stbir__period1_setup_5() stbir__simdf P0, PR; stbir__simdf_load( P0, hc ); stbir__simdf_load1z( PR, hc + 4 );
16231+#define stbir__period1_setup_8() stbir__simdf P0, P1; stbir__simdf_load( P0, hc ); stbir__simdf_load( P1, hc + 4 );
16232+#define stbir__period1_setup_9() stbir__simdf P0, P1, PR; stbir__simdf_load( P0, hc ); stbir__simdf_load( P1, hc + 4 ); stbir__simdf_load1z( PR, hc + 8 );
16233+#define stbir__period1_setup_11() stbir__simdf P0, P1, PR3; stbir__simdf_load( P0, hc ); stbir__simdf_load( P1, hc + 4 ); stbir__simdf_load( PR3, hc + 8 ); // the fourth of PR3 is the 4n+3 zero pad
16234+#define stbir__period1_setup_12() stbir__simdf P0, P1, P2; stbir__simdf_load( P0, hc ); stbir__simdf_load( P1, hc + 4 ); stbir__simdf_load( P2, hc + 8 );
16235+#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1z( PR, hc + 12 );
16236+#define stbir__period1_setup_16() stbir__simdf P3; stbir__period1_setup_12() stbir__simdf_load( P3, hc + 12 );
16237+#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1z( PR, hc + 16 );
16238+
16239+#define stbir__period1_4_coeff_start() \
16240+ stbir__simdf tot, c; \
16241+ STBIR_SIMD_NO_UNROLL(decode); \
16242+ stbir__simdf_mult_mem( tot, P0, decode );
16243+
16244+#define stbir__period1_4_coeff_continue() \
16245+ stbir__simdf_madd_mem( tot, tot, P1, decode + 4 );
16246+#define stbir__period1_4_coeff_continue2() \
16247+ stbir__simdf_madd_mem( tot, tot, P2, decode + 8 );
16248+#define stbir__period1_4_coeff_continue3() \
16249+ stbir__simdf_madd_mem( tot, tot, P3, decode + 12 );
16250+
16251+#define stbir__period1_1_coeff_remnant( ofs ) \
16252+ do { \
16253+ stbir__simdf d; \
16254+ stbir__simdf_load1( d, decode + (ofs) ); \
16255+ stbir__simdf_madd( tot, tot, d, PR ); \
16256+ } while(0)
16257+#define stbir__period1_3_coeff_remnant( ofs ) \
16258+ stbir__simdf_madd_mem( tot, tot, PR3, decode + (ofs) );
16259+
16260+#define stbir__period1_store_output() \
16261+ stbir__simdf_0123to2301( c, tot ); \
16262+ stbir__simdf_add( tot, tot, c ); \
16263+ stbir__simdf_0123to1230( c, tot ); \
16264+ stbir__simdf_add1( tot, tot, c ); \
16265+ stbir__simdf_store1( output, tot ); \
16266+ decode_buffer += advance; \
16267+ output += 1;
16268+
16269+// the four outputs' decode pointers are one base plus multiples of the (constant) advance
16270+#define stbir__period1_x4_setup() \
16271+ float const * d0 = decode_buffer; \
16272+ int const dofs1 = advance, dofs2 = advance * 2, dofs3 = advance * 3; \
16273+ stbir__simdf a0, a1, a2, a3;
16274+
16275+#define stbir__period1_x4_start() \
16276+ do { \
16277+ stbir__simdf_mult_mem( a0, P0, d0 ); \
16278+ stbir__simdf_mult_mem( a1, P0, d0 + dofs1 ); \
16279+ stbir__simdf_mult_mem( a2, P0, d0 + dofs2 ); \
16280+ stbir__simdf_mult_mem( a3, P0, d0 + dofs3 ); \
16281+ } while(0)
16282+
16283+// the next four coefficients, for each of the four outputs
16284+#define stbir__period1_x4_next( P, ofs ) \
16285+ do { \
16286+ stbir__simdf_madd_mem( a0, a0, P, d0 + (ofs) ); \
16287+ stbir__simdf_madd_mem( a1, a1, P, d0 + dofs1 + (ofs) ); \
16288+ stbir__simdf_madd_mem( a2, a2, P, d0 + dofs2 + (ofs) ); \
16289+ stbir__simdf_madd_mem( a3, a3, P, d0 + dofs3 + (ofs) ); \
16290+ } while(0)
16291+#define stbir__period1_x4_continue() stbir__period1_x4_next( P1, 4 )
16292+#define stbir__period1_x4_continue2() stbir__period1_x4_next( P2, 8 )
16293+#define stbir__period1_x4_continue3() stbir__period1_x4_next( P3, 12 )
16294+#define stbir__period1_x4_remnant3( ofs ) stbir__period1_x4_next( PR3, ofs )
16295+
16296+#define stbir__period1_x4_remnant( ofs ) \
16297+ do { \
16298+ stbir__simdf d; \
16299+ stbir__simdf_load1( d, d0 + (ofs) ); \
16300+ stbir__simdf_madd( a0, a0, d, PR ); \
16301+ stbir__simdf_load1( d, d0 + dofs1 + (ofs) ); \
16302+ stbir__simdf_madd( a1, a1, d, PR ); \
16303+ stbir__simdf_load1( d, d0 + dofs2 + (ofs) ); \
16304+ stbir__simdf_madd( a2, a2, d, PR ); \
16305+ stbir__simdf_load1( d, d0 + dofs3 + (ofs) ); \
16306+ stbir__simdf_madd( a3, a3, d, PR ); \
16307+ } while(0)
16308+
16309+#define stbir__period1_x4_store() \
16310+ do { \
16311+ stbir__simdf_addhalves2( a0, a0, a1 ); \
16312+ stbir__simdf_addhalves2( a2, a2, a3 ); \
16313+ stbir__simdf_hadd2( a0, a0, a2 ); \
16314+ stbir__simdf_store( output, a0 ); \
16315+ decode_buffer = d0 + dofs3 + advance; \
16316+ output += 4; \
16317+ } while(0)
16318+
16319+#define stbir__store_output() \
16320+ stbir__simdf_0123to2301( c, tot ); \
16321+ stbir__simdf_add( tot, tot, c ); \
16322+ stbir__simdf_0123to1230( c, tot ); \
16323+ stbir__simdf_add1( tot, tot, c ); \
16324+ stbir__simdf_store1( output, tot ); \
16325+ stbir__next_contributor(); \
16326+ output += 1;
16327+
16328+#else
16329+
16330+#define stbir__1_coeff_only() \
16331+ float tot; \
16332+ tot = decode[0]*hc[0];
16333+
16334+#define stbir__2_coeff_only() \
16335+ float tot; \
16336+ tot = decode[0] * hc[0]; \
16337+ tot += decode[1] * hc[1];
16338+
16339+#define stbir__3_coeff_only() \
16340+ float tot; \
16341+ tot = decode[0] * hc[0]; \
16342+ tot += decode[1] * hc[1]; \
16343+ tot += decode[2] * hc[2];
16344+
16345+#define stbir__store_output_tiny() \
16346+ output[0] = tot; \
16347+ stbir__next_contributor(); \
16348+ output += 1;
16349+
16350+#define stbir__4_coeff_start() \
16351+ float tot0,tot1,tot2,tot3; \
16352+ tot0 = decode[0] * hc[0]; \
16353+ tot1 = decode[1] * hc[1]; \
16354+ tot2 = decode[2] * hc[2]; \
16355+ tot3 = decode[3] * hc[3];
16356+
16357+#define stbir__4_coeff_continue_from_4( ofs ) \
16358+ tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
16359+ tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
16360+ tot2 += decode[2+(ofs)] * hc[2+(ofs)]; \
16361+ tot3 += decode[3+(ofs)] * hc[3+(ofs)];
16362+
16363+#define stbir__1_coeff_remnant( ofs ) \
16364+ tot0 += decode[0+(ofs)] * hc[0+(ofs)];
16365+
16366+#define stbir__2_coeff_remnant( ofs ) \
16367+ tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
16368+ tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
16369+
16370+#define stbir__3_coeff_remnant( ofs ) \
16371+ tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
16372+ tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
16373+ tot2 += decode[2+(ofs)] * hc[2+(ofs)];
16374+
16375+#define stbir__store_output() \
16376+ output[0] = (tot0+tot2)+(tot1+tot3); \
16377+ stbir__next_contributor(); \
16378+ output += 1;
16379+
16380+#endif
16381+
16382+#define STBIR__horizontal_channels 1
16383+#define STB_IMAGE_RESIZE_DO_HORIZONTALS
16384+#include STBIR__HEADER_FILENAME
16385+
16386+
16387+//=================
16388+// Do 2 channel horizontal routines
16389+
16390+#ifdef STBIR_SIMD
16391+
16392+#define stbir__1_coeff_only() \
16393+ stbir__simdf tot,c,d; \
16394+ STBIR_SIMD_NO_UNROLL(decode); \
16395+ stbir__simdf_load1z( c, hc ); \
16396+ stbir__simdf_0123to0011( c, c ); \
16397+ stbir__simdf_load2( d, decode ); \
16398+ stbir__simdf_mult( tot, d, c );
16399+
16400+#define stbir__2_coeff_only() \
16401+ stbir__simdf tot,c; \
16402+ STBIR_SIMD_NO_UNROLL(decode); \
16403+ stbir__simdf_load2( c, hc ); \
16404+ stbir__simdf_0123to0011( c, c ); \
16405+ stbir__simdf_mult_mem( tot, c, decode );
16406+
16407+#define stbir__3_coeff_only() \
16408+ stbir__simdf tot,c,cs,d; \
16409+ STBIR_SIMD_NO_UNROLL(decode); \
16410+ stbir__simdf_load( cs, hc ); \
16411+ stbir__simdf_0123to0011( c, cs ); \
16412+ stbir__simdf_mult_mem( tot, c, decode ); \
16413+ stbir__simdf_0123to2222( c, cs ); \
16414+ stbir__simdf_load2z( d, decode+4 ); \
16415+ stbir__simdf_madd( tot, tot, d, c );
16416+
16417+#define stbir__store_output_tiny() \
16418+ stbir__simdf_0123to2301( c, tot ); \
16419+ stbir__simdf_add( tot, tot, c ); \
16420+ stbir__simdf_store2( output, tot ); \
16421+ stbir__next_contributor(); \
16422+ output += 2;
16423+
16424+// polyphase period 1 for 2 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
16425+#define stbir__period1_setup_4() stbir__simdf P0a, P0b, cs; stbir__simdf_load( cs, hc ); stbir__simdf_0123to0011( P0a, cs ); stbir__simdf_0123to2233( P0b, cs );
16426+#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1z( cs, hc + 4 ); stbir__simdf_0123to0011( PR, cs );
16427+#define stbir__period1_setup_8() stbir__simdf P1a, P1b; stbir__period1_setup_4() stbir__simdf_load( cs, hc + 4 ); stbir__simdf_0123to0011( P1a, cs ); stbir__simdf_0123to2233( P1b, cs );
16428+#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1z( cs, hc + 8 ); stbir__simdf_0123to0011( PR, cs );
16429+#define stbir__period1_setup_11() stbir__simdf PR3a, PR3b; stbir__period1_setup_8() stbir__simdf_load( cs, hc + 8 ); stbir__simdf_0123to0011( PR3a, cs ); stbir__simdf_0123to2222( PR3b, cs );
16430+#define stbir__period1_setup_12() stbir__simdf P2a, P2b; stbir__period1_setup_8() stbir__simdf_load( cs, hc + 8 ); stbir__simdf_0123to0011( P2a, cs ); stbir__simdf_0123to2233( P2b, cs );
16431+#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1z( cs, hc + 12 ); stbir__simdf_0123to0011( PR, cs );
16432+#define stbir__period1_setup_16() stbir__simdf P3a, P3b; stbir__period1_setup_12() stbir__simdf_load( cs, hc + 12 ); stbir__simdf_0123to0011( P3a, cs ); stbir__simdf_0123to2233( P3b, cs );
16433+#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1z( cs, hc + 16 ); stbir__simdf_0123to0011( PR, cs );
16434+
16435+#define stbir__period1_4_coeff_start() \
16436+ stbir__simdf tot0, tot1; \
16437+ STBIR_SIMD_NO_UNROLL(decode); \
16438+ stbir__simdf_mult_mem( tot0, P0a, decode ); \
16439+ stbir__simdf_mult_mem( tot1, P0b, decode + 4 );
16440+
16441+#define stbir__period1_4_coeff_continue() \
16442+ stbir__simdf_madd_mem( tot0, tot0, P1a, decode + 8 ); \
16443+ stbir__simdf_madd_mem( tot1, tot1, P1b, decode + 12 );
16444+#define stbir__period1_4_coeff_continue2() \
16445+ stbir__simdf_madd_mem( tot0, tot0, P2a, decode + 16 ); \
16446+ stbir__simdf_madd_mem( tot1, tot1, P2b, decode + 20 );
16447+#define stbir__period1_4_coeff_continue3() \
16448+ stbir__simdf_madd_mem( tot0, tot0, P3a, decode + 24 ); \
16449+ stbir__simdf_madd_mem( tot1, tot1, P3b, decode + 28 );
16450+
16451+#define stbir__period1_1_coeff_remnant( ofs ) \
16452+ { stbir__simdf d; \
16453+ stbir__simdf_load2( d, decode + (ofs) * 2 ); \
16454+ stbir__simdf_madd( tot0, tot0, d, PR ); }
16455+#define stbir__period1_3_coeff_remnant( ofs ) \
16456+ { stbir__simdf d; \
16457+ stbir__simdf_madd_mem( tot0, tot0, PR3a, decode + (ofs) * 2 ); \
16458+ stbir__simdf_load2z( d, decode + (ofs) * 2 + 4 ); \
16459+ stbir__simdf_madd( tot1, tot1, d, PR3b ); }
16460+
16461+#define stbir__period1_store_output() \
16462+ stbir__simdf_add( tot0, tot0, tot1 ); \
16463+ stbir__simdf_0123to2301( tot1, tot0 ); \
16464+ stbir__simdf_add( tot0, tot0, tot1 ); \
16465+ stbir__simdf_store2( output, tot0 ); \
16466+ decode_buffer += advance; \
16467+ output += 2;
16468+
16469+#ifdef STBIR_SIMD8
16470+
16471+#define stbir__4_coeff_start() \
16472+ stbir__simdf8 tot0,c,cs; \
16473+ STBIR_SIMD_NO_UNROLL(decode); \
16474+ stbir__simdf8_load4b( cs, hc ); \
16475+ stbir__simdf8_0123to00112233( c, cs ); \
16476+ stbir__simdf8_mult_mem( tot0, c, decode );
16477+
16478+#define stbir__4_coeff_continue_from_4( ofs ) \
16479+ STBIR_SIMD_NO_UNROLL(decode); \
16480+ stbir__simdf8_load4b( cs, hc + (ofs) ); \
16481+ stbir__simdf8_0123to00112233( c, cs ); \
16482+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*2 );
16483+
16484+#define stbir__1_coeff_remnant( ofs ) \
16485+ { stbir__simdf t,d; \
16486+ stbir__simdf_load1z( t, hc + (ofs) ); \
16487+ stbir__simdf_load2( d, decode + (ofs) * 2 ); \
16488+ stbir__simdf_0123to0011( t, t ); \
16489+ stbir__simdf_mult( t, t, d ); \
16490+ stbir__simdf8_add4( tot0, tot0, t ); }
16491+
16492+#define stbir__2_coeff_remnant( ofs ) \
16493+ { stbir__simdf t; \
16494+ stbir__simdf_load2( t, hc + (ofs) ); \
16495+ stbir__simdf_0123to0011( t, t ); \
16496+ stbir__simdf_mult_mem( t, t, decode+(ofs)*2 ); \
16497+ stbir__simdf8_add4( tot0, tot0, t ); }
16498+
16499+#define stbir__3_coeff_remnant( ofs ) \
16500+ { stbir__simdf8 d; \
16501+ stbir__simdf8_load4b( cs, hc + (ofs) ); \
16502+ stbir__simdf8_0123to00112233( c, cs ); \
16503+ stbir__simdf8_load6z( d, decode+(ofs)*2 ); \
16504+ stbir__simdf8_madd( tot0, tot0, c, d ); }
16505+
16506+#define stbir__store_output() \
16507+ { stbir__simdf t,d; \
16508+ stbir__simdf8_add4halves( t, stbir__if_simdf8_cast_to_simdf4(tot0), tot0 ); \
16509+ stbir__simdf_0123to2301( d, t ); \
16510+ stbir__simdf_add( t, t, d ); \
16511+ stbir__simdf_store2( output, t ); \
16512+ stbir__next_contributor(); \
16513+ output += 2; }
16514+
16515+#else
16516+
16517+#define stbir__4_coeff_start() \
16518+ stbir__simdf tot0,tot1,c,cs; \
16519+ STBIR_SIMD_NO_UNROLL(decode); \
16520+ stbir__simdf_load( cs, hc ); \
16521+ stbir__simdf_0123to0011( c, cs ); \
16522+ stbir__simdf_mult_mem( tot0, c, decode ); \
16523+ stbir__simdf_0123to2233( c, cs ); \
16524+ stbir__simdf_mult_mem( tot1, c, decode+4 );
16525+
16526+#define stbir__4_coeff_continue_from_4( ofs ) \
16527+ STBIR_SIMD_NO_UNROLL(decode); \
16528+ stbir__simdf_load( cs, hc + (ofs) ); \
16529+ stbir__simdf_0123to0011( c, cs ); \
16530+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 ); \
16531+ stbir__simdf_0123to2233( c, cs ); \
16532+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*2+4 );
16533+
16534+#define stbir__1_coeff_remnant( ofs ) \
16535+ { stbir__simdf d; \
16536+ stbir__simdf_load1z( cs, hc + (ofs) ); \
16537+ stbir__simdf_0123to0011( c, cs ); \
16538+ stbir__simdf_load2( d, decode + (ofs) * 2 ); \
16539+ stbir__simdf_madd( tot0, tot0, d, c ); }
16540+
16541+#define stbir__2_coeff_remnant( ofs ) \
16542+ stbir__simdf_load2( cs, hc + (ofs) ); \
16543+ stbir__simdf_0123to0011( c, cs ); \
16544+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 );
16545+
16546+#define stbir__3_coeff_remnant( ofs ) \
16547+ { stbir__simdf d; \
16548+ stbir__simdf_load( cs, hc + (ofs) ); \
16549+ stbir__simdf_0123to0011( c, cs ); \
16550+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 ); \
16551+ stbir__simdf_0123to2222( c, cs ); \
16552+ stbir__simdf_load2z( d, decode + (ofs) * 2 + 4 ); \
16553+ stbir__simdf_madd( tot1, tot1, d, c ); }
16554+
16555+#define stbir__store_output() \
16556+ stbir__simdf_add( tot0, tot0, tot1 ); \
16557+ stbir__simdf_0123to2301( c, tot0 ); \
16558+ stbir__simdf_add( tot0, tot0, c ); \
16559+ stbir__simdf_store2( output, tot0 ); \
16560+ stbir__next_contributor(); \
16561+ output += 2;
16562+
16563+#endif
16564+
16565+#else
16566+
16567+#define stbir__1_coeff_only() \
16568+ float tota,totb,c; \
16569+ c = hc[0]; \
16570+ tota = decode[0]*c; \
16571+ totb = decode[1]*c;
16572+
16573+#define stbir__2_coeff_only() \
16574+ float tota,totb,c; \
16575+ c = hc[0]; \
16576+ tota = decode[0]*c; \
16577+ totb = decode[1]*c; \
16578+ c = hc[1]; \
16579+ tota += decode[2]*c; \
16580+ totb += decode[3]*c;
16581+
16582+// this weird order of add matches the simd
16583+#define stbir__3_coeff_only() \
16584+ float tota,totb,c; \
16585+ c = hc[0]; \
16586+ tota = decode[0]*c; \
16587+ totb = decode[1]*c; \
16588+ c = hc[2]; \
16589+ tota += decode[4]*c; \
16590+ totb += decode[5]*c; \
16591+ c = hc[1]; \
16592+ tota += decode[2]*c; \
16593+ totb += decode[3]*c;
16594+
16595+#define stbir__store_output_tiny() \
16596+ output[0] = tota; \
16597+ output[1] = totb; \
16598+ stbir__next_contributor(); \
16599+ output += 2;
16600+
16601+#define stbir__4_coeff_start() \
16602+ float tota0,tota1,tota2,tota3,totb0,totb1,totb2,totb3,c; \
16603+ c = hc[0]; \
16604+ tota0 = decode[0]*c; \
16605+ totb0 = decode[1]*c; \
16606+ c = hc[1]; \
16607+ tota1 = decode[2]*c; \
16608+ totb1 = decode[3]*c; \
16609+ c = hc[2]; \
16610+ tota2 = decode[4]*c; \
16611+ totb2 = decode[5]*c; \
16612+ c = hc[3]; \
16613+ tota3 = decode[6]*c; \
16614+ totb3 = decode[7]*c;
16615+
16616+#define stbir__4_coeff_continue_from_4( ofs ) \
16617+ c = hc[0+(ofs)]; \
16618+ tota0 += decode[0+(ofs)*2]*c; \
16619+ totb0 += decode[1+(ofs)*2]*c; \
16620+ c = hc[1+(ofs)]; \
16621+ tota1 += decode[2+(ofs)*2]*c; \
16622+ totb1 += decode[3+(ofs)*2]*c; \
16623+ c = hc[2+(ofs)]; \
16624+ tota2 += decode[4+(ofs)*2]*c; \
16625+ totb2 += decode[5+(ofs)*2]*c; \
16626+ c = hc[3+(ofs)]; \
16627+ tota3 += decode[6+(ofs)*2]*c; \
16628+ totb3 += decode[7+(ofs)*2]*c;
16629+
16630+#define stbir__1_coeff_remnant( ofs ) \
16631+ c = hc[0+(ofs)]; \
16632+ tota0 += decode[0+(ofs)*2] * c; \
16633+ totb0 += decode[1+(ofs)*2] * c;
16634+
16635+#define stbir__2_coeff_remnant( ofs ) \
16636+ c = hc[0+(ofs)]; \
16637+ tota0 += decode[0+(ofs)*2] * c; \
16638+ totb0 += decode[1+(ofs)*2] * c; \
16639+ c = hc[1+(ofs)]; \
16640+ tota1 += decode[2+(ofs)*2] * c; \
16641+ totb1 += decode[3+(ofs)*2] * c;
16642+
16643+#define stbir__3_coeff_remnant( ofs ) \
16644+ c = hc[0+(ofs)]; \
16645+ tota0 += decode[0+(ofs)*2] * c; \
16646+ totb0 += decode[1+(ofs)*2] * c; \
16647+ c = hc[1+(ofs)]; \
16648+ tota1 += decode[2+(ofs)*2] * c; \
16649+ totb1 += decode[3+(ofs)*2] * c; \
16650+ c = hc[2+(ofs)]; \
16651+ tota2 += decode[4+(ofs)*2] * c; \
16652+ totb2 += decode[5+(ofs)*2] * c;
16653+
16654+#define stbir__store_output() \
16655+ output[0] = (tota0+tota2)+(tota1+tota3); \
16656+ output[1] = (totb0+totb2)+(totb1+totb3); \
16657+ stbir__next_contributor(); \
16658+ output += 2;
16659+
16660+#endif
16661+
16662+#define STBIR__horizontal_channels 2
16663+#define STB_IMAGE_RESIZE_DO_HORIZONTALS
16664+#include STBIR__HEADER_FILENAME
16665+
16666+
16667+//=================
16668+// Do 3 channel horizontal routines
16669+
16670+#ifdef STBIR_SIMD
16671+
16672+#define stbir__1_coeff_only() \
16673+ stbir__simdf tot,c,d; \
16674+ STBIR_SIMD_NO_UNROLL(decode); \
16675+ stbir__simdf_load1z( c, hc ); \
16676+ stbir__simdf_0123to0001( c, c ); \
16677+ stbir__simdf_load( d, decode ); \
16678+ stbir__simdf_mult( tot, d, c );
16679+
16680+#define stbir__2_coeff_only() \
16681+ stbir__simdf tot,c,cs,d; \
16682+ STBIR_SIMD_NO_UNROLL(decode); \
16683+ stbir__simdf_load2( cs, hc ); \
16684+ stbir__simdf_0123to0000( c, cs ); \
16685+ stbir__simdf_load( d, decode ); \
16686+ stbir__simdf_mult( tot, d, c ); \
16687+ stbir__simdf_0123to1111( c, cs ); \
16688+ stbir__simdf_load( d, decode+3 ); \
16689+ stbir__simdf_madd( tot, tot, d, c );
16690+
16691+#define stbir__3_coeff_only() \
16692+ stbir__simdf tot,c,d,cs; \
16693+ STBIR_SIMD_NO_UNROLL(decode); \
16694+ stbir__simdf_load( cs, hc ); \
16695+ stbir__simdf_0123to0000( c, cs ); \
16696+ stbir__simdf_load( d, decode ); \
16697+ stbir__simdf_mult( tot, d, c ); \
16698+ stbir__simdf_0123to1111( c, cs ); \
16699+ stbir__simdf_load( d, decode+3 ); \
16700+ stbir__simdf_madd( tot, tot, d, c ); \
16701+ stbir__simdf_0123to2222( c, cs ); \
16702+ stbir__simdf_load( d, decode+6 ); \
16703+ stbir__simdf_madd( tot, tot, d, c );
16704+
16705+#define stbir__store_output_tiny() \
16706+ stbir__simdf_store2( output, tot ); \
16707+ stbir__simdf_0123to2301( tot, tot ); \
16708+ stbir__simdf_store1( output+2, tot ); \
16709+ stbir__next_contributor(); \
16710+ output += 3;
16711+
16712+#ifdef STBIR_SIMD8
16713+
16714+// we're loading from the XXXYYY decode by -1 to get the XXXYYY into different halves of the AVX reg fyi
16715+#define stbir__4_coeff_start() \
16716+ stbir__simdf8 tot0,tot1,c,cs; stbir__simdf t; \
16717+ STBIR_SIMD_NO_UNROLL(decode); \
16718+ stbir__simdf8_load4b( cs, hc ); \
16719+ stbir__simdf8_0123to00001111( c, cs ); \
16720+ stbir__simdf8_mult_mem( tot0, c, decode - 1 ); \
16721+ stbir__simdf8_0123to22223333( c, cs ); \
16722+ stbir__simdf8_mult_mem( tot1, c, decode+6 - 1 );
16723+
16724+#define stbir__4_coeff_continue_from_4( ofs ) \
16725+ STBIR_SIMD_NO_UNROLL(decode); \
16726+ stbir__simdf8_load4b( cs, hc + (ofs) ); \
16727+ stbir__simdf8_0123to00001111( c, cs ); \
16728+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 ); \
16729+ stbir__simdf8_0123to22223333( c, cs ); \
16730+ stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*3 + 6 - 1 );
16731+
16732+#define stbir__1_coeff_remnant( ofs ) \
16733+ STBIR_SIMD_NO_UNROLL(decode); \
16734+ stbir__simdf_load1rep4( t, hc + (ofs) ); \
16735+ stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*3 - 1 );
16736+
16737+#define stbir__2_coeff_remnant( ofs ) \
16738+ STBIR_SIMD_NO_UNROLL(decode); \
16739+ stbir__simdf8_load4b( cs, hc + (ofs) - 2 ); \
16740+ stbir__simdf8_0123to22223333( c, cs ); \
16741+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 );
16742+
16743+ #define stbir__3_coeff_remnant( ofs ) \
16744+ STBIR_SIMD_NO_UNROLL(decode); \
16745+ stbir__simdf8_load4b( cs, hc + (ofs) ); \
16746+ stbir__simdf8_0123to00001111( c, cs ); \
16747+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 ); \
16748+ stbir__simdf8_0123to2222( t, cs ); \
16749+ stbir__simdf8_madd_mem4( tot1, tot1, t, decode+(ofs)*3 + 6 - 1 );
16750+
16751+// polyphase period 1 for 3 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
16752+#define stbir__period1_setup_4() stbir__simdf8 P01, P23, cs; stbir__simdf8_load4b( cs, hc ); stbir__simdf8_0123to00001111( P01, cs ); stbir__simdf8_0123to22223333( P23, cs );
16753+#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1rep4( PR, hc + 4 );
16754+#define stbir__period1_setup_8() stbir__simdf8 P45, P67; stbir__period1_setup_4() stbir__simdf8_load4b( cs, hc + 4 ); stbir__simdf8_0123to00001111( P45, cs ); stbir__simdf8_0123to22223333( P67, cs );
16755+#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1rep4( PR, hc + 8 );
16756+#define stbir__period1_setup_11() stbir__simdf8 P89; stbir__simdf PR; stbir__period1_setup_8() stbir__simdf8_load4b( cs, hc + 8 ); stbir__simdf8_0123to00001111( P89, cs ); stbir__simdf8_0123to2222( PR, cs );
16757+#define stbir__period1_setup_12() stbir__simdf8 P89, PAB; stbir__period1_setup_8() stbir__simdf8_load4b( cs, hc + 8 ); stbir__simdf8_0123to00001111( P89, cs ); stbir__simdf8_0123to22223333( PAB, cs );
16758+#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1rep4( PR, hc + 12 );
16759+#define stbir__period1_setup_16() stbir__simdf8 PCD, PEF; stbir__period1_setup_12() stbir__simdf8_load4b( cs, hc + 12 ); stbir__simdf8_0123to00001111( PCD, cs ); stbir__simdf8_0123to22223333( PEF, cs );
16760+#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1rep4( PR, hc + 16 );
16761+
16762+#define stbir__period1_4_coeff_start() \
16763+ stbir__simdf8 tot0, tot1; stbir__simdf t; \
16764+ STBIR_SIMD_NO_UNROLL(decode); \
16765+ stbir__simdf8_mult_mem( tot0, P01, decode - 1 ); \
16766+ stbir__simdf8_mult_mem( tot1, P23, decode + 6 - 1 );
16767+
16768+#define stbir__period1_4_coeff_continue() \
16769+ stbir__simdf8_madd_mem( tot0, tot0, P45, decode + 12 - 1 ); \
16770+ stbir__simdf8_madd_mem( tot1, tot1, P67, decode + 18 - 1 );
16771+#define stbir__period1_4_coeff_continue2() \
16772+ stbir__simdf8_madd_mem( tot0, tot0, P89, decode + 24 - 1 ); \
16773+ stbir__simdf8_madd_mem( tot1, tot1, PAB, decode + 30 - 1 );
16774+#define stbir__period1_4_coeff_continue3() \
16775+ stbir__simdf8_madd_mem( tot0, tot0, PCD, decode + 36 - 1 ); \
16776+ stbir__simdf8_madd_mem( tot1, tot1, PEF, decode + 42 - 1 );
16777+
16778+#define stbir__period1_1_coeff_remnant( ofs ) \
16779+ stbir__simdf8_madd_mem4( tot0, tot0, PR, decode + (ofs) * 3 - 1 );
16780+#define stbir__period1_3_coeff_remnant( ofs ) \
16781+ stbir__simdf8_madd_mem( tot0, tot0, P89, decode + (ofs) * 3 - 1 ); \
16782+ stbir__simdf8_madd_mem4( tot1, tot1, PR, decode + (ofs) * 3 + 6 - 1 );
16783+
16784+#define stbir__period1_store_output() \
16785+ stbir__simdf8_add( tot0, tot0, tot1 ); \
16786+ stbir__simdf_0123to1230( t, stbir__if_simdf8_cast_to_simdf4( tot0 ) ); \
16787+ stbir__simdf8_add4halves( t, t, tot0 ); \
16788+ decode_buffer += advance; \
16789+ output += 3; \
16790+ if ( output < output_end ) \
16791+ { \
16792+ stbir__simdf_store( output-3, t ); \
16793+ continue; \
16794+ } \
16795+ { stbir__simdf tt; stbir__simdf_0123to2301( tt, t ); \
16796+ stbir__simdf_store2( output-3, t ); \
16797+ stbir__simdf_store1( output+2-3, tt ); } \
16798+ break;
16799+
16800+#define stbir__store_output() \
16801+ stbir__simdf8_add( tot0, tot0, tot1 ); \
16802+ stbir__simdf_0123to1230( t, stbir__if_simdf8_cast_to_simdf4( tot0 ) ); \
16803+ stbir__simdf8_add4halves( t, t, tot0 ); \
16804+ stbir__next_contributor(); \
16805+ output += 3; \
16806+ if ( output < output_end ) \
16807+ { \
16808+ stbir__simdf_store( output-3, t ); \
16809+ continue; \
16810+ } \
16811+ { stbir__simdf tt; stbir__simdf_0123to2301( tt, t ); \
16812+ stbir__simdf_store2( output-3, t ); \
16813+ stbir__simdf_store1( output+2-3, tt ); } \
16814+ break;
16815+
16816+
16817+#else
16818+
16819+#define stbir__4_coeff_start() \
16820+ stbir__simdf tot0,tot1,tot2,c,cs; \
16821+ STBIR_SIMD_NO_UNROLL(decode); \
16822+ stbir__simdf_load( cs, hc ); \
16823+ stbir__simdf_0123to0001( c, cs ); \
16824+ stbir__simdf_mult_mem( tot0, c, decode ); \
16825+ stbir__simdf_0123to1122( c, cs ); \
16826+ stbir__simdf_mult_mem( tot1, c, decode+4 ); \
16827+ stbir__simdf_0123to2333( c, cs ); \
16828+ stbir__simdf_mult_mem( tot2, c, decode+8 );
16829+
16830+#define stbir__4_coeff_continue_from_4( ofs ) \
16831+ STBIR_SIMD_NO_UNROLL(decode); \
16832+ stbir__simdf_load( cs, hc + (ofs) ); \
16833+ stbir__simdf_0123to0001( c, cs ); \
16834+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
16835+ stbir__simdf_0123to1122( c, cs ); \
16836+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*3+4 ); \
16837+ stbir__simdf_0123to2333( c, cs ); \
16838+ stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*3+8 );
16839+
16840+#define stbir__1_coeff_remnant( ofs ) \
16841+ STBIR_SIMD_NO_UNROLL(decode); \
16842+ stbir__simdf_load1z( c, hc + (ofs) ); \
16843+ stbir__simdf_0123to0001( c, c ); \
16844+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 );
16845+
16846+#define stbir__2_coeff_remnant( ofs ) \
16847+ { stbir__simdf d; \
16848+ STBIR_SIMD_NO_UNROLL(decode); \
16849+ stbir__simdf_load2z( cs, hc + (ofs) ); \
16850+ stbir__simdf_0123to0001( c, cs ); \
16851+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
16852+ stbir__simdf_0123to1122( c, cs ); \
16853+ stbir__simdf_load2z( d, decode+(ofs)*3+4 ); \
16854+ stbir__simdf_madd( tot1, tot1, c, d ); }
16855+
16856+#define stbir__3_coeff_remnant( ofs ) \
16857+ { stbir__simdf d; \
16858+ STBIR_SIMD_NO_UNROLL(decode); \
16859+ stbir__simdf_load( cs, hc + (ofs) ); \
16860+ stbir__simdf_0123to0001( c, cs ); \
16861+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
16862+ stbir__simdf_0123to1122( c, cs ); \
16863+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*3+4 ); \
16864+ stbir__simdf_0123to2222( c, cs ); \
16865+ stbir__simdf_load1z( d, decode+(ofs)*3+8 ); \
16866+ stbir__simdf_madd( tot2, tot2, c, d ); }
16867+
16868+// polyphase period 1 for 3 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
16869+#define stbir__period1_setup_4() stbir__simdf P0a, P0b, P0c, cs; stbir__simdf_load( cs, hc ); stbir__simdf_0123to0001( P0a, cs ); stbir__simdf_0123to1122( P0b, cs ); stbir__simdf_0123to2333( P0c, cs );
16870+#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1z( cs, hc + 4 ); stbir__simdf_0123to0001( PR, cs );
16871+#define stbir__period1_setup_8() stbir__simdf P1a, P1b, P1c; stbir__period1_setup_4() stbir__simdf_load( cs, hc + 4 ); stbir__simdf_0123to0001( P1a, cs ); stbir__simdf_0123to1122( P1b, cs ); stbir__simdf_0123to2333( P1c, cs );
16872+#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1z( cs, hc + 8 ); stbir__simdf_0123to0001( PR, cs );
16873+#define stbir__period1_setup_11() stbir__simdf PR3a, PR3b, PR3c; stbir__period1_setup_8() stbir__simdf_load( cs, hc + 8 ); stbir__simdf_0123to0001( PR3a, cs ); stbir__simdf_0123to1122( PR3b, cs ); stbir__simdf_0123to2222( PR3c, cs );
16874+#define stbir__period1_setup_12() stbir__simdf P2a, P2b, P2c; stbir__period1_setup_8() stbir__simdf_load( cs, hc + 8 ); stbir__simdf_0123to0001( P2a, cs ); stbir__simdf_0123to1122( P2b, cs ); stbir__simdf_0123to2333( P2c, cs );
16875+#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1z( cs, hc + 12 ); stbir__simdf_0123to0001( PR, cs );
16876+#define stbir__period1_setup_16() stbir__simdf P3a, P3b, P3c; stbir__period1_setup_12() stbir__simdf_load( cs, hc + 12 ); stbir__simdf_0123to0001( P3a, cs ); stbir__simdf_0123to1122( P3b, cs ); stbir__simdf_0123to2333( P3c, cs );
16877+#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1z( cs, hc + 16 ); stbir__simdf_0123to0001( PR, cs );
16878+
16879+#define stbir__period1_4_coeff_start() \
16880+ stbir__simdf tot0, tot1, tot2, c; \
16881+ STBIR_SIMD_NO_UNROLL(decode); \
16882+ stbir__simdf_mult_mem( tot0, P0a, decode ); \
16883+ stbir__simdf_mult_mem( tot1, P0b, decode + 4 ); \
16884+ stbir__simdf_mult_mem( tot2, P0c, decode + 8 );
16885+
16886+#define stbir__period1_4_coeff_continue() \
16887+ stbir__simdf_madd_mem( tot0, tot0, P1a, decode + 12 ); \
16888+ stbir__simdf_madd_mem( tot1, tot1, P1b, decode + 16 ); \
16889+ stbir__simdf_madd_mem( tot2, tot2, P1c, decode + 20 );
16890+#define stbir__period1_4_coeff_continue2() \
16891+ stbir__simdf_madd_mem( tot0, tot0, P2a, decode + 24 ); \
16892+ stbir__simdf_madd_mem( tot1, tot1, P2b, decode + 28 ); \
16893+ stbir__simdf_madd_mem( tot2, tot2, P2c, decode + 32 );
16894+#define stbir__period1_4_coeff_continue3() \
16895+ stbir__simdf_madd_mem( tot0, tot0, P3a, decode + 36 ); \
16896+ stbir__simdf_madd_mem( tot1, tot1, P3b, decode + 40 ); \
16897+ stbir__simdf_madd_mem( tot2, tot2, P3c, decode + 44 );
16898+
16899+#define stbir__period1_1_coeff_remnant( ofs ) \
16900+ stbir__simdf_madd_mem( tot0, tot0, PR, decode + (ofs) * 3 );
16901+#define stbir__period1_3_coeff_remnant( ofs ) \
16902+ { stbir__simdf d; \
16903+ stbir__simdf_madd_mem( tot0, tot0, PR3a, decode + (ofs) * 3 ); \
16904+ stbir__simdf_madd_mem( tot1, tot1, PR3b, decode + (ofs) * 3 + 4 ); \
16905+ stbir__simdf_load1z( d, decode + (ofs) * 3 + 8 ); \
16906+ stbir__simdf_madd( tot2, tot2, PR3c, d ); }
16907+
16908+#define stbir__period1_store_output() \
16909+ stbir__simdf_0123ABCDto3ABx( c, tot0, tot1 ); \
16910+ stbir__simdf_0123ABCDto23Ax( cs, tot1, tot2 ); \
16911+ stbir__simdf_0123to1230( tot2, tot2 ); \
16912+ stbir__simdf_add( tot0, tot0, cs ); \
16913+ stbir__simdf_add( c, c, tot2 ); \
16914+ stbir__simdf_add( tot0, tot0, c ); \
16915+ decode_buffer += advance; \
16916+ output += 3; \
16917+ if ( output < output_end ) \
16918+ { \
16919+ stbir__simdf_store( output-3, tot0 ); \
16920+ continue; \
16921+ } \
16922+ stbir__simdf_0123to2301( tot1, tot0 ); \
16923+ stbir__simdf_store2( output-3, tot0 ); \
16924+ stbir__simdf_store1( output+2-3, tot1 ); \
16925+ break;
16926+
16927+#define stbir__store_output() \
16928+ stbir__simdf_0123ABCDto3ABx( c, tot0, tot1 ); \
16929+ stbir__simdf_0123ABCDto23Ax( cs, tot1, tot2 ); \
16930+ stbir__simdf_0123to1230( tot2, tot2 ); \
16931+ stbir__simdf_add( tot0, tot0, cs ); \
16932+ stbir__simdf_add( c, c, tot2 ); \
16933+ stbir__simdf_add( tot0, tot0, c ); \
16934+ stbir__next_contributor(); \
16935+ output += 3; \
16936+ if ( output < output_end ) \
16937+ { \
16938+ stbir__simdf_store( output-3, tot0 ); \
16939+ continue; \
16940+ } \
16941+ stbir__simdf_0123to2301( tot1, tot0 ); \
16942+ stbir__simdf_store2( output-3, tot0 ); \
16943+ stbir__simdf_store1( output+2-3, tot1 ); \
16944+ break;
16945+
16946+#endif
16947+
16948+#else
16949+
16950+#define stbir__1_coeff_only() \
16951+ float tot0, tot1, tot2, c; \
16952+ c = hc[0]; \
16953+ tot0 = decode[0]*c; \
16954+ tot1 = decode[1]*c; \
16955+ tot2 = decode[2]*c;
16956+
16957+#define stbir__2_coeff_only() \
16958+ float tot0, tot1, tot2, c; \
16959+ c = hc[0]; \
16960+ tot0 = decode[0]*c; \
16961+ tot1 = decode[1]*c; \
16962+ tot2 = decode[2]*c; \
16963+ c = hc[1]; \
16964+ tot0 += decode[3]*c; \
16965+ tot1 += decode[4]*c; \
16966+ tot2 += decode[5]*c;
16967+
16968+#define stbir__3_coeff_only() \
16969+ float tot0, tot1, tot2, c; \
16970+ c = hc[0]; \
16971+ tot0 = decode[0]*c; \
16972+ tot1 = decode[1]*c; \
16973+ tot2 = decode[2]*c; \
16974+ c = hc[1]; \
16975+ tot0 += decode[3]*c; \
16976+ tot1 += decode[4]*c; \
16977+ tot2 += decode[5]*c; \
16978+ c = hc[2]; \
16979+ tot0 += decode[6]*c; \
16980+ tot1 += decode[7]*c; \
16981+ tot2 += decode[8]*c;
16982+
16983+#define stbir__store_output_tiny() \
16984+ output[0] = tot0; \
16985+ output[1] = tot1; \
16986+ output[2] = tot2; \
16987+ stbir__next_contributor(); \
16988+ output += 3;
16989+
16990+#define stbir__4_coeff_start() \
16991+ float tota0,tota1,tota2,totb0,totb1,totb2,totc0,totc1,totc2,totd0,totd1,totd2,c; \
16992+ c = hc[0]; \
16993+ tota0 = decode[0]*c; \
16994+ tota1 = decode[1]*c; \
16995+ tota2 = decode[2]*c; \
16996+ c = hc[1]; \
16997+ totb0 = decode[3]*c; \
16998+ totb1 = decode[4]*c; \
16999+ totb2 = decode[5]*c; \
17000+ c = hc[2]; \
17001+ totc0 = decode[6]*c; \
17002+ totc1 = decode[7]*c; \
17003+ totc2 = decode[8]*c; \
17004+ c = hc[3]; \
17005+ totd0 = decode[9]*c; \
17006+ totd1 = decode[10]*c; \
17007+ totd2 = decode[11]*c;
17008+
17009+#define stbir__4_coeff_continue_from_4( ofs ) \
17010+ c = hc[0+(ofs)]; \
17011+ tota0 += decode[0+(ofs)*3]*c; \
17012+ tota1 += decode[1+(ofs)*3]*c; \
17013+ tota2 += decode[2+(ofs)*3]*c; \
17014+ c = hc[1+(ofs)]; \
17015+ totb0 += decode[3+(ofs)*3]*c; \
17016+ totb1 += decode[4+(ofs)*3]*c; \
17017+ totb2 += decode[5+(ofs)*3]*c; \
17018+ c = hc[2+(ofs)]; \
17019+ totc0 += decode[6+(ofs)*3]*c; \
17020+ totc1 += decode[7+(ofs)*3]*c; \
17021+ totc2 += decode[8+(ofs)*3]*c; \
17022+ c = hc[3+(ofs)]; \
17023+ totd0 += decode[9+(ofs)*3]*c; \
17024+ totd1 += decode[10+(ofs)*3]*c; \
17025+ totd2 += decode[11+(ofs)*3]*c;
17026+
17027+#define stbir__1_coeff_remnant( ofs ) \
17028+ c = hc[0+(ofs)]; \
17029+ tota0 += decode[0+(ofs)*3]*c; \
17030+ tota1 += decode[1+(ofs)*3]*c; \
17031+ tota2 += decode[2+(ofs)*3]*c;
17032+
17033+#define stbir__2_coeff_remnant( ofs ) \
17034+ c = hc[0+(ofs)]; \
17035+ tota0 += decode[0+(ofs)*3]*c; \
17036+ tota1 += decode[1+(ofs)*3]*c; \
17037+ tota2 += decode[2+(ofs)*3]*c; \
17038+ c = hc[1+(ofs)]; \
17039+ totb0 += decode[3+(ofs)*3]*c; \
17040+ totb1 += decode[4+(ofs)*3]*c; \
17041+ totb2 += decode[5+(ofs)*3]*c; \
17042+
17043+#define stbir__3_coeff_remnant( ofs ) \
17044+ c = hc[0+(ofs)]; \
17045+ tota0 += decode[0+(ofs)*3]*c; \
17046+ tota1 += decode[1+(ofs)*3]*c; \
17047+ tota2 += decode[2+(ofs)*3]*c; \
17048+ c = hc[1+(ofs)]; \
17049+ totb0 += decode[3+(ofs)*3]*c; \
17050+ totb1 += decode[4+(ofs)*3]*c; \
17051+ totb2 += decode[5+(ofs)*3]*c; \
17052+ c = hc[2+(ofs)]; \
17053+ totc0 += decode[6+(ofs)*3]*c; \
17054+ totc1 += decode[7+(ofs)*3]*c; \
17055+ totc2 += decode[8+(ofs)*3]*c;
17056+
17057+#define stbir__store_output() \
17058+ output[0] = (tota0+totc0)+(totb0+totd0); \
17059+ output[1] = (tota1+totc1)+(totb1+totd1); \
17060+ output[2] = (tota2+totc2)+(totb2+totd2); \
17061+ stbir__next_contributor(); \
17062+ output += 3;
17063+
17064+#endif
17065+
17066+#define STBIR__horizontal_channels 3
17067+#define STB_IMAGE_RESIZE_DO_HORIZONTALS
17068+#include STBIR__HEADER_FILENAME
17069+
17070+//=================
17071+// Do 4 channel horizontal routines
17072+
17073+#ifdef STBIR_SIMD
17074+
17075+#define stbir__1_coeff_only() \
17076+ stbir__simdf tot,c; \
17077+ STBIR_SIMD_NO_UNROLL(decode); \
17078+ stbir__simdf_load1( c, hc ); \
17079+ stbir__simdf_0123to0000( c, c ); \
17080+ stbir__simdf_mult_mem( tot, c, decode );
17081+
17082+#define stbir__2_coeff_only() \
17083+ stbir__simdf tot,c,cs; \
17084+ STBIR_SIMD_NO_UNROLL(decode); \
17085+ stbir__simdf_load2( cs, hc ); \
17086+ stbir__simdf_0123to0000( c, cs ); \
17087+ stbir__simdf_mult_mem( tot, c, decode ); \
17088+ stbir__simdf_0123to1111( c, cs ); \
17089+ stbir__simdf_madd_mem( tot, tot, c, decode+4 );
17090+
17091+#define stbir__3_coeff_only() \
17092+ stbir__simdf tot,c,cs; \
17093+ STBIR_SIMD_NO_UNROLL(decode); \
17094+ stbir__simdf_load( cs, hc ); \
17095+ stbir__simdf_0123to0000( c, cs ); \
17096+ stbir__simdf_mult_mem( tot, c, decode ); \
17097+ stbir__simdf_0123to1111( c, cs ); \
17098+ stbir__simdf_madd_mem( tot, tot, c, decode+4 ); \
17099+ stbir__simdf_0123to2222( c, cs ); \
17100+ stbir__simdf_madd_mem( tot, tot, c, decode+8 );
17101+
17102+#define stbir__store_output_tiny() \
17103+ stbir__simdf_store( output, tot ); \
17104+ stbir__next_contributor(); \
17105+ output += 4;
17106+
17107+#ifdef STBIR_SIMD8
17108+
17109+#define stbir__4_coeff_start() \
17110+ stbir__simdf8 tot0,c,cs; stbir__simdf t; \
17111+ STBIR_SIMD_NO_UNROLL(decode); \
17112+ stbir__simdf8_load4b( cs, hc ); \
17113+ stbir__simdf8_0123to00001111( c, cs ); \
17114+ stbir__simdf8_mult_mem( tot0, c, decode ); \
17115+ stbir__simdf8_0123to22223333( c, cs ); \
17116+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+8 );
17117+
17118+#define stbir__4_coeff_continue_from_4( ofs ) \
17119+ STBIR_SIMD_NO_UNROLL(decode); \
17120+ stbir__simdf8_load4b( cs, hc + (ofs) ); \
17121+ stbir__simdf8_0123to00001111( c, cs ); \
17122+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
17123+ stbir__simdf8_0123to22223333( c, cs ); \
17124+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 );
17125+
17126+#define stbir__1_coeff_remnant( ofs ) \
17127+ STBIR_SIMD_NO_UNROLL(decode); \
17128+ stbir__simdf_load1rep4( t, hc + (ofs) ); \
17129+ stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*4 );
17130+
17131+#define stbir__2_coeff_remnant( ofs ) \
17132+ STBIR_SIMD_NO_UNROLL(decode); \
17133+ stbir__simdf8_load4b( cs, hc + (ofs) - 2 ); \
17134+ stbir__simdf8_0123to22223333( c, cs ); \
17135+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 );
17136+
17137+ #define stbir__3_coeff_remnant( ofs ) \
17138+ STBIR_SIMD_NO_UNROLL(decode); \
17139+ stbir__simdf8_load4b( cs, hc + (ofs) ); \
17140+ stbir__simdf8_0123to00001111( c, cs ); \
17141+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
17142+ stbir__simdf8_0123to2222( t, cs ); \
17143+ stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*4+8 );
17144+
17145+// polyphase period 1 for 4 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
17146+#define stbir__period1_setup_4() stbir__simdf8 P01, P23, cs; stbir__simdf8_load4b( cs, hc ); stbir__simdf8_0123to00001111( P01, cs ); stbir__simdf8_0123to22223333( P23, cs );
17147+#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1rep4( PR, hc + 4 );
17148+#define stbir__period1_setup_8() stbir__simdf8 P45, P67; stbir__period1_setup_4() stbir__simdf8_load4b( cs, hc + 4 ); stbir__simdf8_0123to00001111( P45, cs ); stbir__simdf8_0123to22223333( P67, cs );
17149+#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1rep4( PR, hc + 8 );
17150+#define stbir__period1_setup_11() stbir__simdf8 P89; stbir__simdf PR; stbir__period1_setup_8() stbir__simdf8_load4b( cs, hc + 8 ); stbir__simdf8_0123to00001111( P89, cs ); stbir__simdf8_0123to2222( PR, cs );
17151+#define stbir__period1_setup_12() stbir__simdf8 P89, PAB; stbir__period1_setup_8() stbir__simdf8_load4b( cs, hc + 8 ); stbir__simdf8_0123to00001111( P89, cs ); stbir__simdf8_0123to22223333( PAB, cs );
17152+#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1rep4( PR, hc + 12 );
17153+#define stbir__period1_setup_16() stbir__simdf8 PCD, PEF; stbir__period1_setup_12() stbir__simdf8_load4b( cs, hc + 12 ); stbir__simdf8_0123to00001111( PCD, cs ); stbir__simdf8_0123to22223333( PEF, cs );
17154+#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1rep4( PR, hc + 16 );
17155+
17156+#define stbir__period1_4_coeff_start() \
17157+ stbir__simdf8 tot0; stbir__simdf t; \
17158+ STBIR_SIMD_NO_UNROLL(decode); \
17159+ stbir__simdf8_mult_mem( tot0, P01, decode ); \
17160+ stbir__simdf8_madd_mem( tot0, tot0, P23, decode + 8 );
17161+
17162+#define stbir__period1_4_coeff_continue() \
17163+ stbir__simdf8_madd_mem( tot0, tot0, P45, decode + 16 ); \
17164+ stbir__simdf8_madd_mem( tot0, tot0, P67, decode + 24 );
17165+#define stbir__period1_4_coeff_continue2() \
17166+ stbir__simdf8_madd_mem( tot0, tot0, P89, decode + 32 ); \
17167+ stbir__simdf8_madd_mem( tot0, tot0, PAB, decode + 40 );
17168+#define stbir__period1_4_coeff_continue3() \
17169+ stbir__simdf8_madd_mem( tot0, tot0, PCD, decode + 48 ); \
17170+ stbir__simdf8_madd_mem( tot0, tot0, PEF, decode + 56 );
17171+
17172+#define stbir__period1_1_coeff_remnant( ofs ) \
17173+ stbir__simdf8_madd_mem4( tot0, tot0, PR, decode + (ofs) * 4 );
17174+#define stbir__period1_3_coeff_remnant( ofs ) \
17175+ stbir__simdf8_madd_mem( tot0, tot0, P89, decode + (ofs) * 4 ); \
17176+ stbir__simdf8_madd_mem4( tot0, tot0, PR, decode + (ofs) * 4 + 8 );
17177+
17178+#define stbir__period1_store_output() \
17179+ stbir__simdf8_add4halves( t, stbir__if_simdf8_cast_to_simdf4(tot0), tot0 ); \
17180+ stbir__simdf_store( output, t ); \
17181+ decode_buffer += advance; \
17182+ output += 4;
17183+
17184+#define stbir__store_output() \
17185+ stbir__simdf8_add4halves( t, stbir__if_simdf8_cast_to_simdf4(tot0), tot0 ); \
17186+ stbir__simdf_store( output, t ); \
17187+ stbir__next_contributor(); \
17188+ output += 4;
17189+
17190+#else
17191+
17192+#define stbir__4_coeff_start() \
17193+ stbir__simdf tot0,tot1,c,cs; \
17194+ STBIR_SIMD_NO_UNROLL(decode); \
17195+ stbir__simdf_load( cs, hc ); \
17196+ stbir__simdf_0123to0000( c, cs ); \
17197+ stbir__simdf_mult_mem( tot0, c, decode ); \
17198+ stbir__simdf_0123to1111( c, cs ); \
17199+ stbir__simdf_mult_mem( tot1, c, decode+4 ); \
17200+ stbir__simdf_0123to2222( c, cs ); \
17201+ stbir__simdf_madd_mem( tot0, tot0, c, decode+8 ); \
17202+ stbir__simdf_0123to3333( c, cs ); \
17203+ stbir__simdf_madd_mem( tot1, tot1, c, decode+12 );
17204+
17205+#define stbir__4_coeff_continue_from_4( ofs ) \
17206+ STBIR_SIMD_NO_UNROLL(decode); \
17207+ stbir__simdf_load( cs, hc + (ofs) ); \
17208+ stbir__simdf_0123to0000( c, cs ); \
17209+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
17210+ stbir__simdf_0123to1111( c, cs ); \
17211+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 ); \
17212+ stbir__simdf_0123to2222( c, cs ); \
17213+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 ); \
17214+ stbir__simdf_0123to3333( c, cs ); \
17215+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+12 );
17216+
17217+#define stbir__1_coeff_remnant( ofs ) \
17218+ STBIR_SIMD_NO_UNROLL(decode); \
17219+ stbir__simdf_load1( c, hc + (ofs) ); \
17220+ stbir__simdf_0123to0000( c, c ); \
17221+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 );
17222+
17223+#define stbir__2_coeff_remnant( ofs ) \
17224+ STBIR_SIMD_NO_UNROLL(decode); \
17225+ stbir__simdf_load2( cs, hc + (ofs) ); \
17226+ stbir__simdf_0123to0000( c, cs ); \
17227+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
17228+ stbir__simdf_0123to1111( c, cs ); \
17229+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 );
17230+
17231+#define stbir__3_coeff_remnant( ofs ) \
17232+ STBIR_SIMD_NO_UNROLL(decode); \
17233+ stbir__simdf_load( cs, hc + (ofs) ); \
17234+ stbir__simdf_0123to0000( c, cs ); \
17235+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
17236+ stbir__simdf_0123to1111( c, cs ); \
17237+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 ); \
17238+ stbir__simdf_0123to2222( c, cs ); \
17239+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 );
17240+
17241+// polyphase period 1 for 4 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
17242+#define stbir__period1_setup_4() stbir__simdf P0, P1, P2, P3; stbir__simdf_load1frep4( P0, hc[0] ); stbir__simdf_load1frep4( P1, hc[1] ); stbir__simdf_load1frep4( P2, hc[2] ); stbir__simdf_load1frep4( P3, hc[3] );
17243+#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1frep4( PR, hc[4] );
17244+#define stbir__period1_setup_8() stbir__simdf P4, P5, P6, P7; stbir__period1_setup_4() stbir__simdf_load1frep4( P4, hc[4] ); stbir__simdf_load1frep4( P5, hc[5] ); stbir__simdf_load1frep4( P6, hc[6] ); stbir__simdf_load1frep4( P7, hc[7] );
17245+#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1frep4( PR, hc[8] );
17246+#define stbir__period1_setup_11() stbir__simdf P8, P9, P10; stbir__period1_setup_8() stbir__simdf_load1frep4( P8, hc[8] ); stbir__simdf_load1frep4( P9, hc[9] ); stbir__simdf_load1frep4( P10, hc[10] );
17247+#define stbir__period1_setup_12() stbir__simdf P8, P9, P10, P11; stbir__period1_setup_8() stbir__simdf_load1frep4( P8, hc[8] ); stbir__simdf_load1frep4( P9, hc[9] ); stbir__simdf_load1frep4( P10, hc[10] ); stbir__simdf_load1frep4( P11, hc[11] );
17248+#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1frep4( PR, hc[12] );
17249+#define stbir__period1_setup_16() stbir__simdf P12, P13, P14, P15; stbir__period1_setup_12() stbir__simdf_load1frep4( P12, hc[12] ); stbir__simdf_load1frep4( P13, hc[13] ); stbir__simdf_load1frep4( P14, hc[14] ); stbir__simdf_load1frep4( P15, hc[15] );
17250+#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1frep4( PR, hc[16] );
17251+
17252+#define stbir__period1_4_coeff_start() \
17253+ stbir__simdf tot0, tot1; \
17254+ STBIR_SIMD_NO_UNROLL(decode); \
17255+ stbir__simdf_mult_mem( tot0, P0, decode ); \
17256+ stbir__simdf_mult_mem( tot1, P1, decode + 4 ); \
17257+ stbir__simdf_madd_mem( tot0, tot0, P2, decode + 8 ); \
17258+ stbir__simdf_madd_mem( tot1, tot1, P3, decode + 12 );
17259+
17260+#define stbir__period1_4_coeff_continue() \
17261+ stbir__simdf_madd_mem( tot0, tot0, P4, decode + 16 ); \
17262+ stbir__simdf_madd_mem( tot1, tot1, P5, decode + 20 ); \
17263+ stbir__simdf_madd_mem( tot0, tot0, P6, decode + 24 ); \
17264+ stbir__simdf_madd_mem( tot1, tot1, P7, decode + 28 );
17265+#define stbir__period1_4_coeff_continue2() \
17266+ stbir__simdf_madd_mem( tot0, tot0, P8, decode + 32 ); \
17267+ stbir__simdf_madd_mem( tot1, tot1, P9, decode + 36 ); \
17268+ stbir__simdf_madd_mem( tot0, tot0, P10, decode + 40 ); \
17269+ stbir__simdf_madd_mem( tot1, tot1, P11, decode + 44 );
17270+#define stbir__period1_4_coeff_continue3() \
17271+ stbir__simdf_madd_mem( tot0, tot0, P12, decode + 48 ); \
17272+ stbir__simdf_madd_mem( tot1, tot1, P13, decode + 52 ); \
17273+ stbir__simdf_madd_mem( tot0, tot0, P14, decode + 56 ); \
17274+ stbir__simdf_madd_mem( tot1, tot1, P15, decode + 60 );
17275+
17276+#define stbir__period1_1_coeff_remnant( ofs ) \
17277+ stbir__simdf_madd_mem( tot0, tot0, PR, decode + (ofs) * 4 );
17278+#define stbir__period1_3_coeff_remnant( ofs ) \
17279+ stbir__simdf_madd_mem( tot0, tot0, P8, decode + (ofs) * 4 ); \
17280+ stbir__simdf_madd_mem( tot1, tot1, P9, decode + (ofs) * 4 + 4 ); \
17281+ stbir__simdf_madd_mem( tot0, tot0, P10, decode + (ofs) * 4 + 8 );
17282+
17283+#define stbir__period1_store_output() \
17284+ stbir__simdf_add( tot0, tot0, tot1 ); \
17285+ stbir__simdf_store( output, tot0 ); \
17286+ decode_buffer += advance; \
17287+ output += 4;
17288+
17289+#define stbir__store_output() \
17290+ stbir__simdf_add( tot0, tot0, tot1 ); \
17291+ stbir__simdf_store( output, tot0 ); \
17292+ stbir__next_contributor(); \
17293+ output += 4;
17294+
17295+#endif
17296+
17297+#else
17298+
17299+#define stbir__1_coeff_only() \
17300+ float p0,p1,p2,p3,c; \
17301+ STBIR_SIMD_NO_UNROLL(decode); \
17302+ c = hc[0]; \
17303+ p0 = decode[0] * c; \
17304+ p1 = decode[1] * c; \
17305+ p2 = decode[2] * c; \
17306+ p3 = decode[3] * c;
17307+
17308+#define stbir__2_coeff_only() \
17309+ float p0,p1,p2,p3,c; \
17310+ STBIR_SIMD_NO_UNROLL(decode); \
17311+ c = hc[0]; \
17312+ p0 = decode[0] * c; \
17313+ p1 = decode[1] * c; \
17314+ p2 = decode[2] * c; \
17315+ p3 = decode[3] * c; \
17316+ c = hc[1]; \
17317+ p0 += decode[4] * c; \
17318+ p1 += decode[5] * c; \
17319+ p2 += decode[6] * c; \
17320+ p3 += decode[7] * c;
17321+
17322+#define stbir__3_coeff_only() \
17323+ float p0,p1,p2,p3,c; \
17324+ STBIR_SIMD_NO_UNROLL(decode); \
17325+ c = hc[0]; \
17326+ p0 = decode[0] * c; \
17327+ p1 = decode[1] * c; \
17328+ p2 = decode[2] * c; \
17329+ p3 = decode[3] * c; \
17330+ c = hc[1]; \
17331+ p0 += decode[4] * c; \
17332+ p1 += decode[5] * c; \
17333+ p2 += decode[6] * c; \
17334+ p3 += decode[7] * c; \
17335+ c = hc[2]; \
17336+ p0 += decode[8] * c; \
17337+ p1 += decode[9] * c; \
17338+ p2 += decode[10] * c; \
17339+ p3 += decode[11] * c;
17340+
17341+#define stbir__store_output_tiny() \
17342+ output[0] = p0; \
17343+ output[1] = p1; \
17344+ output[2] = p2; \
17345+ output[3] = p3; \
17346+ stbir__next_contributor(); \
17347+ output += 4;
17348+
17349+#define stbir__4_coeff_start() \
17350+ float x0,x1,x2,x3,y0,y1,y2,y3,c; \
17351+ STBIR_SIMD_NO_UNROLL(decode); \
17352+ c = hc[0]; \
17353+ x0 = decode[0] * c; \
17354+ x1 = decode[1] * c; \
17355+ x2 = decode[2] * c; \
17356+ x3 = decode[3] * c; \
17357+ c = hc[1]; \
17358+ y0 = decode[4] * c; \
17359+ y1 = decode[5] * c; \
17360+ y2 = decode[6] * c; \
17361+ y3 = decode[7] * c; \
17362+ c = hc[2]; \
17363+ x0 += decode[8] * c; \
17364+ x1 += decode[9] * c; \
17365+ x2 += decode[10] * c; \
17366+ x3 += decode[11] * c; \
17367+ c = hc[3]; \
17368+ y0 += decode[12] * c; \
17369+ y1 += decode[13] * c; \
17370+ y2 += decode[14] * c; \
17371+ y3 += decode[15] * c;
17372+
17373+#define stbir__4_coeff_continue_from_4( ofs ) \
17374+ STBIR_SIMD_NO_UNROLL(decode); \
17375+ c = hc[0+(ofs)]; \
17376+ x0 += decode[0+(ofs)*4] * c; \
17377+ x1 += decode[1+(ofs)*4] * c; \
17378+ x2 += decode[2+(ofs)*4] * c; \
17379+ x3 += decode[3+(ofs)*4] * c; \
17380+ c = hc[1+(ofs)]; \
17381+ y0 += decode[4+(ofs)*4] * c; \
17382+ y1 += decode[5+(ofs)*4] * c; \
17383+ y2 += decode[6+(ofs)*4] * c; \
17384+ y3 += decode[7+(ofs)*4] * c; \
17385+ c = hc[2+(ofs)]; \
17386+ x0 += decode[8+(ofs)*4] * c; \
17387+ x1 += decode[9+(ofs)*4] * c; \
17388+ x2 += decode[10+(ofs)*4] * c; \
17389+ x3 += decode[11+(ofs)*4] * c; \
17390+ c = hc[3+(ofs)]; \
17391+ y0 += decode[12+(ofs)*4] * c; \
17392+ y1 += decode[13+(ofs)*4] * c; \
17393+ y2 += decode[14+(ofs)*4] * c; \
17394+ y3 += decode[15+(ofs)*4] * c;
17395+
17396+#define stbir__1_coeff_remnant( ofs ) \
17397+ STBIR_SIMD_NO_UNROLL(decode); \
17398+ c = hc[0+(ofs)]; \
17399+ x0 += decode[0+(ofs)*4] * c; \
17400+ x1 += decode[1+(ofs)*4] * c; \
17401+ x2 += decode[2+(ofs)*4] * c; \
17402+ x3 += decode[3+(ofs)*4] * c;
17403+
17404+#define stbir__2_coeff_remnant( ofs ) \
17405+ STBIR_SIMD_NO_UNROLL(decode); \
17406+ c = hc[0+(ofs)]; \
17407+ x0 += decode[0+(ofs)*4] * c; \
17408+ x1 += decode[1+(ofs)*4] * c; \
17409+ x2 += decode[2+(ofs)*4] * c; \
17410+ x3 += decode[3+(ofs)*4] * c; \
17411+ c = hc[1+(ofs)]; \
17412+ y0 += decode[4+(ofs)*4] * c; \
17413+ y1 += decode[5+(ofs)*4] * c; \
17414+ y2 += decode[6+(ofs)*4] * c; \
17415+ y3 += decode[7+(ofs)*4] * c;
17416+
17417+#define stbir__3_coeff_remnant( ofs ) \
17418+ STBIR_SIMD_NO_UNROLL(decode); \
17419+ c = hc[0+(ofs)]; \
17420+ x0 += decode[0+(ofs)*4] * c; \
17421+ x1 += decode[1+(ofs)*4] * c; \
17422+ x2 += decode[2+(ofs)*4] * c; \
17423+ x3 += decode[3+(ofs)*4] * c; \
17424+ c = hc[1+(ofs)]; \
17425+ y0 += decode[4+(ofs)*4] * c; \
17426+ y1 += decode[5+(ofs)*4] * c; \
17427+ y2 += decode[6+(ofs)*4] * c; \
17428+ y3 += decode[7+(ofs)*4] * c; \
17429+ c = hc[2+(ofs)]; \
17430+ x0 += decode[8+(ofs)*4] * c; \
17431+ x1 += decode[9+(ofs)*4] * c; \
17432+ x2 += decode[10+(ofs)*4] * c; \
17433+ x3 += decode[11+(ofs)*4] * c;
17434+
17435+#define stbir__store_output() \
17436+ output[0] = x0 + y0; \
17437+ output[1] = x1 + y1; \
17438+ output[2] = x2 + y2; \
17439+ output[3] = x3 + y3; \
17440+ stbir__next_contributor(); \
17441+ output += 4;
17442+
17443+#endif
17444+
17445+#define STBIR__horizontal_channels 4
17446+#define STB_IMAGE_RESIZE_DO_HORIZONTALS
17447+#include STBIR__HEADER_FILENAME
17448+
17449+
17450+//=================
17451+// Do 7 channel horizontal routines
17452+
17453+#ifdef STBIR_SIMD
17454+
17455+#define stbir__1_coeff_only() \
17456+ stbir__simdf tot0,tot1,c; \
17457+ STBIR_SIMD_NO_UNROLL(decode); \
17458+ stbir__simdf_load1( c, hc ); \
17459+ stbir__simdf_0123to0000( c, c ); \
17460+ stbir__simdf_mult_mem( tot0, c, decode ); \
17461+ stbir__simdf_mult_mem( tot1, c, decode+3 );
17462+
17463+#define stbir__2_coeff_only() \
17464+ stbir__simdf tot0,tot1,c,cs; \
17465+ STBIR_SIMD_NO_UNROLL(decode); \
17466+ stbir__simdf_load2( cs, hc ); \
17467+ stbir__simdf_0123to0000( c, cs ); \
17468+ stbir__simdf_mult_mem( tot0, c, decode ); \
17469+ stbir__simdf_mult_mem( tot1, c, decode+3 ); \
17470+ stbir__simdf_0123to1111( c, cs ); \
17471+ stbir__simdf_madd_mem( tot0, tot0, c, decode+7 ); \
17472+ stbir__simdf_madd_mem( tot1, tot1, c,decode+10 );
17473+
17474+#define stbir__3_coeff_only() \
17475+ stbir__simdf tot0,tot1,c,cs; \
17476+ STBIR_SIMD_NO_UNROLL(decode); \
17477+ stbir__simdf_load( cs, hc ); \
17478+ stbir__simdf_0123to0000( c, cs ); \
17479+ stbir__simdf_mult_mem( tot0, c, decode ); \
17480+ stbir__simdf_mult_mem( tot1, c, decode+3 ); \
17481+ stbir__simdf_0123to1111( c, cs ); \
17482+ stbir__simdf_madd_mem( tot0, tot0, c, decode+7 ); \
17483+ stbir__simdf_madd_mem( tot1, tot1, c, decode+10 ); \
17484+ stbir__simdf_0123to2222( c, cs ); \
17485+ stbir__simdf_madd_mem( tot0, tot0, c, decode+14 ); \
17486+ stbir__simdf_madd_mem( tot1, tot1, c, decode+17 );
17487+
17488+#define stbir__store_output_tiny() \
17489+ stbir__simdf_store( output+3, tot1 ); \
17490+ stbir__simdf_store( output, tot0 ); \
17491+ stbir__next_contributor(); \
17492+ output += 7;
17493+
17494+#ifdef STBIR_SIMD8
17495+
17496+#define stbir__4_coeff_start() \
17497+ stbir__simdf8 tot0,tot1,c,cs; \
17498+ STBIR_SIMD_NO_UNROLL(decode); \
17499+ stbir__simdf8_load4b( cs, hc ); \
17500+ stbir__simdf8_0123to00000000( c, cs ); \
17501+ stbir__simdf8_mult_mem( tot0, c, decode ); \
17502+ stbir__simdf8_0123to11111111( c, cs ); \
17503+ stbir__simdf8_mult_mem( tot1, c, decode+7 ); \
17504+ stbir__simdf8_0123to22222222( c, cs ); \
17505+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+14 ); \
17506+ stbir__simdf8_0123to33333333( c, cs ); \
17507+ stbir__simdf8_madd_mem( tot1, tot1, c, decode+21 );
17508+
17509+#define stbir__4_coeff_continue_from_4( ofs ) \
17510+ STBIR_SIMD_NO_UNROLL(decode); \
17511+ stbir__simdf8_load4b( cs, hc + (ofs) ); \
17512+ stbir__simdf8_0123to00000000( c, cs ); \
17513+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
17514+ stbir__simdf8_0123to11111111( c, cs ); \
17515+ stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 ); \
17516+ stbir__simdf8_0123to22222222( c, cs ); \
17517+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
17518+ stbir__simdf8_0123to33333333( c, cs ); \
17519+ stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+21 );
17520+
17521+#define stbir__1_coeff_remnant( ofs ) \
17522+ STBIR_SIMD_NO_UNROLL(decode); \
17523+ stbir__simdf8_load1b( c, hc + (ofs) ); \
17524+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 );
17525+
17526+#define stbir__2_coeff_remnant( ofs ) \
17527+ STBIR_SIMD_NO_UNROLL(decode); \
17528+ stbir__simdf8_load1b( c, hc + (ofs) ); \
17529+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
17530+ stbir__simdf8_load1b( c, hc + (ofs)+1 ); \
17531+ stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 );
17532+
17533+#define stbir__3_coeff_remnant( ofs ) \
17534+ STBIR_SIMD_NO_UNROLL(decode); \
17535+ stbir__simdf8_load4b( cs, hc + (ofs) ); \
17536+ stbir__simdf8_0123to00000000( c, cs ); \
17537+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
17538+ stbir__simdf8_0123to11111111( c, cs ); \
17539+ stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 ); \
17540+ stbir__simdf8_0123to22222222( c, cs ); \
17541+ stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 );
17542+
17543+// polyphase period 1 for 7 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
17544+#define stbir__period1_setup_4() stbir__simdf8 P0, P1, P2, P3; stbir__simdf8_load1b( P0, hc ); stbir__simdf8_load1b( P1, hc + 1 ); stbir__simdf8_load1b( P2, hc + 2 ); stbir__simdf8_load1b( P3, hc + 3 );
17545+#define stbir__period1_setup_5() stbir__simdf8 PR; stbir__period1_setup_4() stbir__simdf8_load1b( PR, hc + 4 );
17546+#define stbir__period1_setup_8() stbir__simdf8 P4, P5, P6, P7; stbir__period1_setup_4() stbir__simdf8_load1b( P4, hc + 4 ); stbir__simdf8_load1b( P5, hc + 5 ); stbir__simdf8_load1b( P6, hc + 6 ); stbir__simdf8_load1b( P7, hc + 7 );
17547+#define stbir__period1_setup_9() stbir__simdf8 PR; stbir__period1_setup_8() stbir__simdf8_load1b( PR, hc + 8 );
17548+#define stbir__period1_setup_11() stbir__simdf8 P8, P9, P10; stbir__period1_setup_8() stbir__simdf8_load1b( P8, hc + 8 ); stbir__simdf8_load1b( P9, hc + 9 ); stbir__simdf8_load1b( P10, hc + 10 );
17549+#define stbir__period1_setup_12() stbir__simdf8 P8, P9, P10, P11; stbir__period1_setup_8() stbir__simdf8_load1b( P8, hc + 8 ); stbir__simdf8_load1b( P9, hc + 9 ); stbir__simdf8_load1b( P10, hc + 10 ); stbir__simdf8_load1b( P11, hc + 11 );
17550+#define stbir__period1_setup_13() stbir__simdf8 PR; stbir__period1_setup_12() stbir__simdf8_load1b( PR, hc + 12 );
17551+#define stbir__period1_setup_16() stbir__simdf8 P12, P13, P14, P15; stbir__period1_setup_12() stbir__simdf8_load1b( P12, hc + 12 ); stbir__simdf8_load1b( P13, hc + 13 ); stbir__simdf8_load1b( P14, hc + 14 ); stbir__simdf8_load1b( P15, hc + 15 );
17552+#define stbir__period1_setup_17() stbir__simdf8 PR; stbir__period1_setup_16() stbir__simdf8_load1b( PR, hc + 16 );
17553+
17554+#define stbir__period1_4_coeff_start() \
17555+ stbir__simdf8 tot0, tot1; \
17556+ STBIR_SIMD_NO_UNROLL(decode); \
17557+ stbir__simdf8_mult_mem( tot0, P0, decode ); \
17558+ stbir__simdf8_mult_mem( tot1, P1, decode + 7 ); \
17559+ stbir__simdf8_madd_mem( tot0, tot0, P2, decode + 14 ); \
17560+ stbir__simdf8_madd_mem( tot1, tot1, P3, decode + 21 );
17561+
17562+#define stbir__period1_4_coeff_continue() \
17563+ stbir__simdf8_madd_mem( tot0, tot0, P4, decode + 28 ); \
17564+ stbir__simdf8_madd_mem( tot1, tot1, P5, decode + 35 ); \
17565+ stbir__simdf8_madd_mem( tot0, tot0, P6, decode + 42 ); \
17566+ stbir__simdf8_madd_mem( tot1, tot1, P7, decode + 49 );
17567+#define stbir__period1_4_coeff_continue2() \
17568+ stbir__simdf8_madd_mem( tot0, tot0, P8, decode + 56 ); \
17569+ stbir__simdf8_madd_mem( tot1, tot1, P9, decode + 63 ); \
17570+ stbir__simdf8_madd_mem( tot0, tot0, P10, decode + 70 ); \
17571+ stbir__simdf8_madd_mem( tot1, tot1, P11, decode + 77 );
17572+#define stbir__period1_4_coeff_continue3() \
17573+ stbir__simdf8_madd_mem( tot0, tot0, P12, decode + 84 ); \
17574+ stbir__simdf8_madd_mem( tot1, tot1, P13, decode + 91 ); \
17575+ stbir__simdf8_madd_mem( tot0, tot0, P14, decode + 98 ); \
17576+ stbir__simdf8_madd_mem( tot1, tot1, P15, decode + 105 );
17577+
17578+#define stbir__period1_1_coeff_remnant( ofs ) \
17579+ stbir__simdf8_madd_mem( tot0, tot0, PR, decode + (ofs) * 7 );
17580+#define stbir__period1_3_coeff_remnant( ofs ) \
17581+ stbir__simdf8_madd_mem( tot0, tot0, P8, decode + (ofs) * 7 ); \
17582+ stbir__simdf8_madd_mem( tot1, tot1, P9, decode + (ofs) * 7 + 7 ); \
17583+ stbir__simdf8_madd_mem( tot0, tot0, P10, decode + (ofs) * 7 + 14 );
17584+
17585+#define stbir__period1_store_output() \
17586+ stbir__simdf8_add( tot0, tot0, tot1 ); \
17587+ decode_buffer += advance; \
17588+ output += 7; \
17589+ if ( output < output_end ) \
17590+ { \
17591+ stbir__simdf8_store( output-7, tot0 ); \
17592+ continue; \
17593+ } \
17594+ stbir__simdf_store( output-7+3, stbir__simdf_swiz(stbir__simdf8_gettop4(tot0),0,0,1,2) ); \
17595+ stbir__simdf_store( output-7, stbir__if_simdf8_cast_to_simdf4(tot0) ); \
17596+ break;
17597+
17598+#define stbir__store_output() \
17599+ stbir__simdf8_add( tot0, tot0, tot1 ); \
17600+ stbir__next_contributor(); \
17601+ output += 7; \
17602+ if ( output < output_end ) \
17603+ { \
17604+ stbir__simdf8_store( output-7, tot0 ); \
17605+ continue; \
17606+ } \
17607+ stbir__simdf_store( output-7+3, stbir__simdf_swiz(stbir__simdf8_gettop4(tot0),0,0,1,2) ); \
17608+ stbir__simdf_store( output-7, stbir__if_simdf8_cast_to_simdf4(tot0) ); \
17609+ break;
17610+
17611+#else
17612+
17613+#define stbir__4_coeff_start() \
17614+ stbir__simdf tot0,tot1,tot2,tot3,c,cs; \
17615+ STBIR_SIMD_NO_UNROLL(decode); \
17616+ stbir__simdf_load( cs, hc ); \
17617+ stbir__simdf_0123to0000( c, cs ); \
17618+ stbir__simdf_mult_mem( tot0, c, decode ); \
17619+ stbir__simdf_mult_mem( tot1, c, decode+3 ); \
17620+ stbir__simdf_0123to1111( c, cs ); \
17621+ stbir__simdf_mult_mem( tot2, c, decode+7 ); \
17622+ stbir__simdf_mult_mem( tot3, c, decode+10 ); \
17623+ stbir__simdf_0123to2222( c, cs ); \
17624+ stbir__simdf_madd_mem( tot0, tot0, c, decode+14 ); \
17625+ stbir__simdf_madd_mem( tot1, tot1, c, decode+17 ); \
17626+ stbir__simdf_0123to3333( c, cs ); \
17627+ stbir__simdf_madd_mem( tot2, tot2, c, decode+21 ); \
17628+ stbir__simdf_madd_mem( tot3, tot3, c, decode+24 );
17629+
17630+#define stbir__4_coeff_continue_from_4( ofs ) \
17631+ STBIR_SIMD_NO_UNROLL(decode); \
17632+ stbir__simdf_load( cs, hc + (ofs) ); \
17633+ stbir__simdf_0123to0000( c, cs ); \
17634+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
17635+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
17636+ stbir__simdf_0123to1111( c, cs ); \
17637+ stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
17638+ stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 ); \
17639+ stbir__simdf_0123to2222( c, cs ); \
17640+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
17641+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+17 ); \
17642+ stbir__simdf_0123to3333( c, cs ); \
17643+ stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+21 ); \
17644+ stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+24 );
17645+
17646+#define stbir__1_coeff_remnant( ofs ) \
17647+ STBIR_SIMD_NO_UNROLL(decode); \
17648+ stbir__simdf_load1( c, hc + (ofs) ); \
17649+ stbir__simdf_0123to0000( c, c ); \
17650+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
17651+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
17652+
17653+#define stbir__2_coeff_remnant( ofs ) \
17654+ STBIR_SIMD_NO_UNROLL(decode); \
17655+ stbir__simdf_load2( cs, hc + (ofs) ); \
17656+ stbir__simdf_0123to0000( c, cs ); \
17657+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
17658+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
17659+ stbir__simdf_0123to1111( c, cs ); \
17660+ stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
17661+ stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 );
17662+
17663+#define stbir__3_coeff_remnant( ofs ) \
17664+ STBIR_SIMD_NO_UNROLL(decode); \
17665+ stbir__simdf_load( cs, hc + (ofs) ); \
17666+ stbir__simdf_0123to0000( c, cs ); \
17667+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
17668+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
17669+ stbir__simdf_0123to1111( c, cs ); \
17670+ stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
17671+ stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 ); \
17672+ stbir__simdf_0123to2222( c, cs ); \
17673+ stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
17674+ stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+17 );
17675+
17676+// polyphase period 1 for 7 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
17677+#define stbir__period1_setup_4() stbir__simdf P0, P1, P2, P3; stbir__simdf_load1frep4( P0, hc[0] ); stbir__simdf_load1frep4( P1, hc[1] ); stbir__simdf_load1frep4( P2, hc[2] ); stbir__simdf_load1frep4( P3, hc[3] );
17678+#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1frep4( PR, hc[4] );
17679+#define stbir__period1_setup_8() stbir__simdf P4, P5, P6, P7; stbir__period1_setup_4() stbir__simdf_load1frep4( P4, hc[4] ); stbir__simdf_load1frep4( P5, hc[5] ); stbir__simdf_load1frep4( P6, hc[6] ); stbir__simdf_load1frep4( P7, hc[7] );
17680+#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1frep4( PR, hc[8] );
17681+#define stbir__period1_setup_11() stbir__simdf P8, P9, P10; stbir__period1_setup_8() stbir__simdf_load1frep4( P8, hc[8] ); stbir__simdf_load1frep4( P9, hc[9] ); stbir__simdf_load1frep4( P10, hc[10] );
17682+#define stbir__period1_setup_12() stbir__simdf P8, P9, P10, P11; stbir__period1_setup_8() stbir__simdf_load1frep4( P8, hc[8] ); stbir__simdf_load1frep4( P9, hc[9] ); stbir__simdf_load1frep4( P10, hc[10] ); stbir__simdf_load1frep4( P11, hc[11] );
17683+#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1frep4( PR, hc[12] );
17684+#define stbir__period1_setup_16() stbir__simdf P12, P13, P14, P15; stbir__period1_setup_12() stbir__simdf_load1frep4( P12, hc[12] ); stbir__simdf_load1frep4( P13, hc[13] ); stbir__simdf_load1frep4( P14, hc[14] ); stbir__simdf_load1frep4( P15, hc[15] );
17685+#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1frep4( PR, hc[16] );
17686+
17687+#define stbir__period1_4_coeff_start() \
17688+ stbir__simdf tot0, tot1, tot2, tot3; \
17689+ STBIR_SIMD_NO_UNROLL(decode); \
17690+ stbir__simdf_mult_mem( tot0, P0, decode ); \
17691+ stbir__simdf_mult_mem( tot1, P0, decode + 3 ); \
17692+ stbir__simdf_mult_mem( tot2, P1, decode + 7 ); \
17693+ stbir__simdf_mult_mem( tot3, P1, decode + 10 ); \
17694+ stbir__simdf_madd_mem( tot0, tot0, P2, decode + 14 ); \
17695+ stbir__simdf_madd_mem( tot1, tot1, P2, decode + 17 ); \
17696+ stbir__simdf_madd_mem( tot2, tot2, P3, decode + 21 ); \
17697+ stbir__simdf_madd_mem( tot3, tot3, P3, decode + 24 );
17698+
17699+#define stbir__period1_4_coeff_continue() \
17700+ stbir__simdf_madd_mem( tot0, tot0, P4, decode + 28 ); \
17701+ stbir__simdf_madd_mem( tot1, tot1, P4, decode + 31 ); \
17702+ stbir__simdf_madd_mem( tot2, tot2, P5, decode + 35 ); \
17703+ stbir__simdf_madd_mem( tot3, tot3, P5, decode + 38 ); \
17704+ stbir__simdf_madd_mem( tot0, tot0, P6, decode + 42 ); \
17705+ stbir__simdf_madd_mem( tot1, tot1, P6, decode + 45 ); \
17706+ stbir__simdf_madd_mem( tot2, tot2, P7, decode + 49 ); \
17707+ stbir__simdf_madd_mem( tot3, tot3, P7, decode + 52 );
17708+#define stbir__period1_4_coeff_continue2() \
17709+ stbir__simdf_madd_mem( tot0, tot0, P8, decode + 56 ); \
17710+ stbir__simdf_madd_mem( tot1, tot1, P8, decode + 59 ); \
17711+ stbir__simdf_madd_mem( tot2, tot2, P9, decode + 63 ); \
17712+ stbir__simdf_madd_mem( tot3, tot3, P9, decode + 66 ); \
17713+ stbir__simdf_madd_mem( tot0, tot0, P10, decode + 70 ); \
17714+ stbir__simdf_madd_mem( tot1, tot1, P10, decode + 73 ); \
17715+ stbir__simdf_madd_mem( tot2, tot2, P11, decode + 77 ); \
17716+ stbir__simdf_madd_mem( tot3, tot3, P11, decode + 80 );
17717+#define stbir__period1_4_coeff_continue3() \
17718+ stbir__simdf_madd_mem( tot0, tot0, P12, decode + 84 ); \
17719+ stbir__simdf_madd_mem( tot1, tot1, P12, decode + 87 ); \
17720+ stbir__simdf_madd_mem( tot2, tot2, P13, decode + 91 ); \
17721+ stbir__simdf_madd_mem( tot3, tot3, P13, decode + 94 ); \
17722+ stbir__simdf_madd_mem( tot0, tot0, P14, decode + 98 ); \
17723+ stbir__simdf_madd_mem( tot1, tot1, P14, decode + 101 ); \
17724+ stbir__simdf_madd_mem( tot2, tot2, P15, decode + 105 ); \
17725+ stbir__simdf_madd_mem( tot3, tot3, P15, decode + 108 );
17726+
17727+#define stbir__period1_1_coeff_remnant( ofs ) \
17728+ stbir__simdf_madd_mem( tot0, tot0, PR, decode + (ofs) * 7 ); \
17729+ stbir__simdf_madd_mem( tot1, tot1, PR, decode + (ofs) * 7 + 3 );
17730+#define stbir__period1_3_coeff_remnant( ofs ) \
17731+ stbir__simdf_madd_mem( tot0, tot0, P8, decode + (ofs) * 7 ); \
17732+ stbir__simdf_madd_mem( tot1, tot1, P8, decode + (ofs) * 7 + 3 ); \
17733+ stbir__simdf_madd_mem( tot2, tot2, P9, decode + (ofs) * 7 + 7 ); \
17734+ stbir__simdf_madd_mem( tot3, tot3, P9, decode + (ofs) * 7 + 10 ); \
17735+ stbir__simdf_madd_mem( tot0, tot0, P10, decode + (ofs) * 7 + 14 ); \
17736+ stbir__simdf_madd_mem( tot1, tot1, P10, decode + (ofs) * 7 + 17 );
17737+
17738+#define stbir__period1_store_output() \
17739+ stbir__simdf_add( tot0, tot0, tot2 ); \
17740+ stbir__simdf_add( tot1, tot1, tot3 ); \
17741+ stbir__simdf_store( output+3, tot1 ); \
17742+ stbir__simdf_store( output, tot0 ); \
17743+ decode_buffer += advance; \
17744+ output += 7;
17745+
17746+#define stbir__store_output() \
17747+ stbir__simdf_add( tot0, tot0, tot2 ); \
17748+ stbir__simdf_add( tot1, tot1, tot3 ); \
17749+ stbir__simdf_store( output+3, tot1 ); \
17750+ stbir__simdf_store( output, tot0 ); \
17751+ stbir__next_contributor(); \
17752+ output += 7;
17753+
17754+#endif
17755+
17756+#else
17757+
17758+#define stbir__1_coeff_only() \
17759+ float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
17760+ c = hc[0]; \
17761+ tot0 = decode[0]*c; \
17762+ tot1 = decode[1]*c; \
17763+ tot2 = decode[2]*c; \
17764+ tot3 = decode[3]*c; \
17765+ tot4 = decode[4]*c; \
17766+ tot5 = decode[5]*c; \
17767+ tot6 = decode[6]*c;
17768+
17769+#define stbir__2_coeff_only() \
17770+ float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
17771+ c = hc[0]; \
17772+ tot0 = decode[0]*c; \
17773+ tot1 = decode[1]*c; \
17774+ tot2 = decode[2]*c; \
17775+ tot3 = decode[3]*c; \
17776+ tot4 = decode[4]*c; \
17777+ tot5 = decode[5]*c; \
17778+ tot6 = decode[6]*c; \
17779+ c = hc[1]; \
17780+ tot0 += decode[7]*c; \
17781+ tot1 += decode[8]*c; \
17782+ tot2 += decode[9]*c; \
17783+ tot3 += decode[10]*c; \
17784+ tot4 += decode[11]*c; \
17785+ tot5 += decode[12]*c; \
17786+ tot6 += decode[13]*c; \
17787+
17788+#define stbir__3_coeff_only() \
17789+ float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
17790+ c = hc[0]; \
17791+ tot0 = decode[0]*c; \
17792+ tot1 = decode[1]*c; \
17793+ tot2 = decode[2]*c; \
17794+ tot3 = decode[3]*c; \
17795+ tot4 = decode[4]*c; \
17796+ tot5 = decode[5]*c; \
17797+ tot6 = decode[6]*c; \
17798+ c = hc[1]; \
17799+ tot0 += decode[7]*c; \
17800+ tot1 += decode[8]*c; \
17801+ tot2 += decode[9]*c; \
17802+ tot3 += decode[10]*c; \
17803+ tot4 += decode[11]*c; \
17804+ tot5 += decode[12]*c; \
17805+ tot6 += decode[13]*c; \
17806+ c = hc[2]; \
17807+ tot0 += decode[14]*c; \
17808+ tot1 += decode[15]*c; \
17809+ tot2 += decode[16]*c; \
17810+ tot3 += decode[17]*c; \
17811+ tot4 += decode[18]*c; \
17812+ tot5 += decode[19]*c; \
17813+ tot6 += decode[20]*c; \
17814+
17815+#define stbir__store_output_tiny() \
17816+ output[0] = tot0; \
17817+ output[1] = tot1; \
17818+ output[2] = tot2; \
17819+ output[3] = tot3; \
17820+ output[4] = tot4; \
17821+ output[5] = tot5; \
17822+ output[6] = tot6; \
17823+ stbir__next_contributor(); \
17824+ output += 7;
17825+
17826+#define stbir__4_coeff_start() \
17827+ float x0,x1,x2,x3,x4,x5,x6,y0,y1,y2,y3,y4,y5,y6,c; \
17828+ STBIR_SIMD_NO_UNROLL(decode); \
17829+ c = hc[0]; \
17830+ x0 = decode[0] * c; \
17831+ x1 = decode[1] * c; \
17832+ x2 = decode[2] * c; \
17833+ x3 = decode[3] * c; \
17834+ x4 = decode[4] * c; \
17835+ x5 = decode[5] * c; \
17836+ x6 = decode[6] * c; \
17837+ c = hc[1]; \
17838+ y0 = decode[7] * c; \
17839+ y1 = decode[8] * c; \
17840+ y2 = decode[9] * c; \
17841+ y3 = decode[10] * c; \
17842+ y4 = decode[11] * c; \
17843+ y5 = decode[12] * c; \
17844+ y6 = decode[13] * c; \
17845+ c = hc[2]; \
17846+ x0 += decode[14] * c; \
17847+ x1 += decode[15] * c; \
17848+ x2 += decode[16] * c; \
17849+ x3 += decode[17] * c; \
17850+ x4 += decode[18] * c; \
17851+ x5 += decode[19] * c; \
17852+ x6 += decode[20] * c; \
17853+ c = hc[3]; \
17854+ y0 += decode[21] * c; \
17855+ y1 += decode[22] * c; \
17856+ y2 += decode[23] * c; \
17857+ y3 += decode[24] * c; \
17858+ y4 += decode[25] * c; \
17859+ y5 += decode[26] * c; \
17860+ y6 += decode[27] * c;
17861+
17862+#define stbir__4_coeff_continue_from_4( ofs ) \
17863+ STBIR_SIMD_NO_UNROLL(decode); \
17864+ c = hc[0+(ofs)]; \
17865+ x0 += decode[0+(ofs)*7] * c; \
17866+ x1 += decode[1+(ofs)*7] * c; \
17867+ x2 += decode[2+(ofs)*7] * c; \
17868+ x3 += decode[3+(ofs)*7] * c; \
17869+ x4 += decode[4+(ofs)*7] * c; \
17870+ x5 += decode[5+(ofs)*7] * c; \
17871+ x6 += decode[6+(ofs)*7] * c; \
17872+ c = hc[1+(ofs)]; \
17873+ y0 += decode[7+(ofs)*7] * c; \
17874+ y1 += decode[8+(ofs)*7] * c; \
17875+ y2 += decode[9+(ofs)*7] * c; \
17876+ y3 += decode[10+(ofs)*7] * c; \
17877+ y4 += decode[11+(ofs)*7] * c; \
17878+ y5 += decode[12+(ofs)*7] * c; \
17879+ y6 += decode[13+(ofs)*7] * c; \
17880+ c = hc[2+(ofs)]; \
17881+ x0 += decode[14+(ofs)*7] * c; \
17882+ x1 += decode[15+(ofs)*7] * c; \
17883+ x2 += decode[16+(ofs)*7] * c; \
17884+ x3 += decode[17+(ofs)*7] * c; \
17885+ x4 += decode[18+(ofs)*7] * c; \
17886+ x5 += decode[19+(ofs)*7] * c; \
17887+ x6 += decode[20+(ofs)*7] * c; \
17888+ c = hc[3+(ofs)]; \
17889+ y0 += decode[21+(ofs)*7] * c; \
17890+ y1 += decode[22+(ofs)*7] * c; \
17891+ y2 += decode[23+(ofs)*7] * c; \
17892+ y3 += decode[24+(ofs)*7] * c; \
17893+ y4 += decode[25+(ofs)*7] * c; \
17894+ y5 += decode[26+(ofs)*7] * c; \
17895+ y6 += decode[27+(ofs)*7] * c;
17896+
17897+#define stbir__1_coeff_remnant( ofs ) \
17898+ STBIR_SIMD_NO_UNROLL(decode); \
17899+ c = hc[0+(ofs)]; \
17900+ x0 += decode[0+(ofs)*7] * c; \
17901+ x1 += decode[1+(ofs)*7] * c; \
17902+ x2 += decode[2+(ofs)*7] * c; \
17903+ x3 += decode[3+(ofs)*7] * c; \
17904+ x4 += decode[4+(ofs)*7] * c; \
17905+ x5 += decode[5+(ofs)*7] * c; \
17906+ x6 += decode[6+(ofs)*7] * c; \
17907+
17908+#define stbir__2_coeff_remnant( ofs ) \
17909+ STBIR_SIMD_NO_UNROLL(decode); \
17910+ c = hc[0+(ofs)]; \
17911+ x0 += decode[0+(ofs)*7] * c; \
17912+ x1 += decode[1+(ofs)*7] * c; \
17913+ x2 += decode[2+(ofs)*7] * c; \
17914+ x3 += decode[3+(ofs)*7] * c; \
17915+ x4 += decode[4+(ofs)*7] * c; \
17916+ x5 += decode[5+(ofs)*7] * c; \
17917+ x6 += decode[6+(ofs)*7] * c; \
17918+ c = hc[1+(ofs)]; \
17919+ y0 += decode[7+(ofs)*7] * c; \
17920+ y1 += decode[8+(ofs)*7] * c; \
17921+ y2 += decode[9+(ofs)*7] * c; \
17922+ y3 += decode[10+(ofs)*7] * c; \
17923+ y4 += decode[11+(ofs)*7] * c; \
17924+ y5 += decode[12+(ofs)*7] * c; \
17925+ y6 += decode[13+(ofs)*7] * c; \
17926+
17927+#define stbir__3_coeff_remnant( ofs ) \
17928+ STBIR_SIMD_NO_UNROLL(decode); \
17929+ c = hc[0+(ofs)]; \
17930+ x0 += decode[0+(ofs)*7] * c; \
17931+ x1 += decode[1+(ofs)*7] * c; \
17932+ x2 += decode[2+(ofs)*7] * c; \
17933+ x3 += decode[3+(ofs)*7] * c; \
17934+ x4 += decode[4+(ofs)*7] * c; \
17935+ x5 += decode[5+(ofs)*7] * c; \
17936+ x6 += decode[6+(ofs)*7] * c; \
17937+ c = hc[1+(ofs)]; \
17938+ y0 += decode[7+(ofs)*7] * c; \
17939+ y1 += decode[8+(ofs)*7] * c; \
17940+ y2 += decode[9+(ofs)*7] * c; \
17941+ y3 += decode[10+(ofs)*7] * c; \
17942+ y4 += decode[11+(ofs)*7] * c; \
17943+ y5 += decode[12+(ofs)*7] * c; \
17944+ y6 += decode[13+(ofs)*7] * c; \
17945+ c = hc[2+(ofs)]; \
17946+ x0 += decode[14+(ofs)*7] * c; \
17947+ x1 += decode[15+(ofs)*7] * c; \
17948+ x2 += decode[16+(ofs)*7] * c; \
17949+ x3 += decode[17+(ofs)*7] * c; \
17950+ x4 += decode[18+(ofs)*7] * c; \
17951+ x5 += decode[19+(ofs)*7] * c; \
17952+ x6 += decode[20+(ofs)*7] * c; \
17953+
17954+#define stbir__store_output() \
17955+ output[0] = x0 + y0; \
17956+ output[1] = x1 + y1; \
17957+ output[2] = x2 + y2; \
17958+ output[3] = x3 + y3; \
17959+ output[4] = x4 + y4; \
17960+ output[5] = x5 + y5; \
17961+ output[6] = x6 + y6; \
17962+ stbir__next_contributor(); \
17963+ output += 7;
17964+
17965+#endif
17966+
17967+#define STBIR__horizontal_channels 7
17968+#define STB_IMAGE_RESIZE_DO_HORIZONTALS
17969+#include STBIR__HEADER_FILENAME
17970+
17971+
17972+// include all of the vertical resamplers (both scatter and gather versions)
17973+
17974+#define STBIR__vertical_channels 1
17975+#define STB_IMAGE_RESIZE_DO_VERTICALS
17976+#include STBIR__HEADER_FILENAME
17977+
17978+#define STBIR__vertical_channels 1
17979+#define STB_IMAGE_RESIZE_DO_VERTICALS
17980+#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
17981+#include STBIR__HEADER_FILENAME
17982+
17983+#define STBIR__vertical_channels 2
17984+#define STB_IMAGE_RESIZE_DO_VERTICALS
17985+#include STBIR__HEADER_FILENAME
17986+
17987+#define STBIR__vertical_channels 2
17988+#define STB_IMAGE_RESIZE_DO_VERTICALS
17989+#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
17990+#include STBIR__HEADER_FILENAME
17991+
17992+#define STBIR__vertical_channels 3
17993+#define STB_IMAGE_RESIZE_DO_VERTICALS
17994+#include STBIR__HEADER_FILENAME
17995+
17996+#define STBIR__vertical_channels 3
17997+#define STB_IMAGE_RESIZE_DO_VERTICALS
17998+#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
17999+#include STBIR__HEADER_FILENAME
18000+
18001+#define STBIR__vertical_channels 4
18002+#define STB_IMAGE_RESIZE_DO_VERTICALS
18003+#include STBIR__HEADER_FILENAME
18004+
18005+#define STBIR__vertical_channels 4
18006+#define STB_IMAGE_RESIZE_DO_VERTICALS
18007+#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
18008+#include STBIR__HEADER_FILENAME
18009+
18010+#define STBIR__vertical_channels 5
18011+#define STB_IMAGE_RESIZE_DO_VERTICALS
18012+#include STBIR__HEADER_FILENAME
18013+
18014+#define STBIR__vertical_channels 5
18015+#define STB_IMAGE_RESIZE_DO_VERTICALS
18016+#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
18017+#include STBIR__HEADER_FILENAME
18018+
18019+#define STBIR__vertical_channels 6
18020+#define STB_IMAGE_RESIZE_DO_VERTICALS
18021+#include STBIR__HEADER_FILENAME
18022+
18023+#define STBIR__vertical_channels 6
18024+#define STB_IMAGE_RESIZE_DO_VERTICALS
18025+#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
18026+#include STBIR__HEADER_FILENAME
18027+
18028+#define STBIR__vertical_channels 7
18029+#define STB_IMAGE_RESIZE_DO_VERTICALS
18030+#include STBIR__HEADER_FILENAME
18031+
18032+#define STBIR__vertical_channels 7
18033+#define STB_IMAGE_RESIZE_DO_VERTICALS
18034+#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
18035+#include STBIR__HEADER_FILENAME
18036+
18037+#define STBIR__vertical_channels 8
18038+#define STB_IMAGE_RESIZE_DO_VERTICALS
18039+#include STBIR__HEADER_FILENAME
18040+
18041+#define STBIR__vertical_channels 8
18042+#define STB_IMAGE_RESIZE_DO_VERTICALS
18043+#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
18044+#include STBIR__HEADER_FILENAME
18045+
18046+typedef void STBIR_VERTICAL_GATHERFUNC( float * output, float const * coeffs, float const ** inputs, float const * input0_end );
18047+
18048+static STBIR_VERTICAL_GATHERFUNC * stbir__vertical_gathers[ 8 ] =
18049+{
18050+ stbir__vertical_gather_with_1_coeffs,stbir__vertical_gather_with_2_coeffs,stbir__vertical_gather_with_3_coeffs,stbir__vertical_gather_with_4_coeffs,stbir__vertical_gather_with_5_coeffs,stbir__vertical_gather_with_6_coeffs,stbir__vertical_gather_with_7_coeffs,stbir__vertical_gather_with_8_coeffs
18051+};
18052+
18053+static STBIR_VERTICAL_GATHERFUNC * stbir__vertical_gathers_continues[ 8 ] =
18054+{
18055+ stbir__vertical_gather_with_1_coeffs_cont,stbir__vertical_gather_with_2_coeffs_cont,stbir__vertical_gather_with_3_coeffs_cont,stbir__vertical_gather_with_4_coeffs_cont,stbir__vertical_gather_with_5_coeffs_cont,stbir__vertical_gather_with_6_coeffs_cont,stbir__vertical_gather_with_7_coeffs_cont,stbir__vertical_gather_with_8_coeffs_cont
18056+};
18057+
18058+typedef void STBIR_VERTICAL_SCATTERFUNC( float ** outputs, float const * coeffs, float const * input, float const * input_end );
18059+
18060+static STBIR_VERTICAL_SCATTERFUNC * stbir__vertical_scatter_sets[ 8 ] =
18061+{
18062+ stbir__vertical_scatter_with_1_coeffs,stbir__vertical_scatter_with_2_coeffs,stbir__vertical_scatter_with_3_coeffs,stbir__vertical_scatter_with_4_coeffs,stbir__vertical_scatter_with_5_coeffs,stbir__vertical_scatter_with_6_coeffs,stbir__vertical_scatter_with_7_coeffs,stbir__vertical_scatter_with_8_coeffs
18063+};
18064+
18065+static STBIR_VERTICAL_SCATTERFUNC * stbir__vertical_scatter_blends[ 8 ] =
18066+{
18067+ stbir__vertical_scatter_with_1_coeffs_cont,stbir__vertical_scatter_with_2_coeffs_cont,stbir__vertical_scatter_with_3_coeffs_cont,stbir__vertical_scatter_with_4_coeffs_cont,stbir__vertical_scatter_with_5_coeffs_cont,stbir__vertical_scatter_with_6_coeffs_cont,stbir__vertical_scatter_with_7_coeffs_cont,stbir__vertical_scatter_with_8_coeffs_cont
18068+};
18069+
18070+
18071+// pointer to an output pixel, relative to the output region (offset_x/offset_y are already folded into output_data)
18072+static void * stbir__output_pixel_ptr( stbir__info const * stbir_info, int x, int y )
18073+{
18074+ return ( (char*) stbir_info->output_data ) + (ptrdiff_t)( y * stbir_info->output_stride_bytes ) + ( (size_t)x * (size_t)stbir_info->channels * (size_t)stbir__type_size[ stbir_info->output_type ] );
18075+}
18076+
18077+static void stbir__encode_scanline( stbir__info const * stbir_info, float * encode_buffer, int row, stbir__column_slice const * slice STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
18078+{
18079+ int num_pixels = slice->width;
18080+ int channels = stbir_info->channels;
18081+ ptrdiff_t width_times_channels = (ptrdiff_t)num_pixels * channels;
18082+ void * output_buffer_data = stbir__output_pixel_ptr( stbir_info, slice->x0, row );
18083+ void * output_buffer;
18084+
18085+ // un-alpha weight if we need to
18086+ if ( stbir_info->alpha_unweight )
18087+ {
18088+ STBIR_PROFILE_START( unalpha );
18089+ stbir_info->alpha_unweight( encode_buffer, width_times_channels );
18090+ STBIR_PROFILE_END( unalpha );
18091+ }
18092+
18093+ // write directly into output by default
18094+ output_buffer = output_buffer_data;
18095+
18096+ // if we have an output callback, we first convert the decode buffer in place (and then hand that to the callback)
18097+ if ( stbir_info->out_pixels_cb )
18098+ output_buffer = encode_buffer;
18099+
18100+ STBIR_PROFILE_START( encode );
18101+ // convert into the output buffer
18102+ stbir_info->encode_pixels( output_buffer, width_times_channels, encode_buffer );
18103+ STBIR_PROFILE_END( encode );
18104+
18105+ // if we have an output callback, call it to send the data (x and y are in full output image space, not region relative)
18106+ if ( stbir_info->out_pixels_cb )
18107+ stbir_info->out_pixels_cb( output_buffer, num_pixels, stbir_info->offset_x + slice->x0, stbir_info->offset_y + row, stbir_info->user_data );
18108+}
18109+
18110+
18111+// Get the ring buffer pointer for an index
18112+static float* stbir__get_ring_buffer_entry(stbir__info const * stbir_info, stbir__per_split_info const * split_info, int index )
18113+{
18114+ STBIR_ASSERT( index < stbir_info->ring_buffer_num_entries );
18115+
18116+ #ifdef STBIR__SEPARATE_ALLOCATIONS
18117+ return split_info->ring_buffers[ index ];
18118+ #else
18119+ return (float*) ( ( (char*) split_info->ring_buffer ) + ( (size_t) index * stbir_info->ring_buffer_length_bytes ) );
18120+ #endif
18121+}
18122+
18123+// Get the specified scan line from the ring buffer
18124+static float* stbir__get_ring_buffer_scanline(stbir__info const * stbir_info, stbir__per_split_info const * split_info, int get_scanline)
18125+{
18126+ int ring_buffer_index = (split_info->ring_buffer_begin_index + (get_scanline - split_info->ring_buffer_first_scanline)) % stbir_info->ring_buffer_num_entries;
18127+ return stbir__get_ring_buffer_entry( stbir_info, split_info, ring_buffer_index );
18128+}
18129+
18130+// the horizontal pass over one slice's output pixels.
18131+static void stbir__resample_horizontal_gather(stbir__info const * stbir_info, float* output_buffer, float const * input_buffer, stbir__column_slice const * slice STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
18132+{
18133+ stbir__sampler const * horizontal = &stbir_info->horizontal;
18134+ int channels = stbir_info->effective_channels;
18135+ int point_copy = ( horizontal->filter_enum == STBIR_FILTER_POINT_SAMPLE ) && ( stbir__scale_is_copy( &horizontal->scale_info ) );
18136+ int prologue = horizontal->shared_prologue;
18137+ int period = horizontal->shared_period;
18138+ int epilogue_start = horizontal->num_contributors - horizontal->shared_epilogue;
18139+ int x = slice->x0, x_end = slice->x0 + slice->width;
18140+ int n0 = slice->first_n0;
18141+
18142+ STBIR_PROFILE_START( horizontal );
18143+ while ( x < x_end )
18144+ {
18145+ float const * decode = input_buffer + (ptrdiff_t)( n0 - slice->in_n0 ) * channels; // input_buffer holds input pixel in_n0 at index zero
18146+ int run_end = x_end;
18147+ int entry_stride = horizontal->entry_floats * (int) sizeof( float );
18148+
18149+ if ( ( period ) && ( x < epilogue_start ) )
18150+ {
18151+ if ( period == 1 )
18152+ {
18153+ // one set for the whole periodic region: the prologue is its own run, then the region is one run over that set
18154+ run_end = ( x < prologue ) ? prologue : epilogue_start;
18155+ if ( x >= prologue )
18156+ entry_stride = 0;
18157+ }
18158+ else
18159+ {
18160+ // to the end of this pass over the block (from the prologue, the first pass), but not into the epilogue
18161+ int pass = ( x < prologue ) ? 0 : ( ( x - prologue ) / period );
18162+ run_end = prologue + ( pass + 1 ) * period;
18163+ if ( run_end > epilogue_start )
18164+ run_end = epilogue_start;
18165+ }
18166+ if ( run_end > x_end )
18167+ run_end = x_end;
18168+ }
18169+
18170+ if ( point_copy )
18171+ STBIR_MEMCPY( output_buffer, decode, ( run_end - x ) * sizeof( float ) * channels );
18172+ else if ( ( entry_stride == 0 ) && ( stbir_info->horizontal_gather_period1 ) )
18173+ stbir_info->horizontal_gather_period1( output_buffer, run_end - x, decode, stbir__output_column_entry( horizontal, x ), 0 );
18174+ else
18175+ stbir_info->horizontal_gather_channels( output_buffer, run_end - x, decode, stbir__output_column_entry( horizontal, x ), entry_stride );
18176+ output_buffer += (ptrdiff_t)( run_end - x ) * channels;
18177+
18178+ // where the next run starts reading: the epilogue, or a block boundary
18179+ if ( run_end == epilogue_start )
18180+ n0 = horizontal->shared_epilogue_n0;
18181+ else if ( period )
18182+ n0 = horizontal->shared_block_n0 + ( ( run_end - prologue ) / period ) * horizontal->shared_block_shift;
18183+ x = run_end;
18184+ }
18185+ STBIR_PROFILE_END( horizontal );
18186+}
18187+
18188+// how many floats wide the buffer that the vertical pass works on is (decoded input pixels when vertical
18189+// first, horizontally resampled output pixels otherwise)
18190+static ptrdiff_t stbir__vertical_width_times_channels( stbir__info const * stbir_info, stbir__column_slice const * slice )
18191+{
18192+ int width = ( stbir_info->vertical_first ) ? ( slice->in_n1 - slice->in_n0 + 1 ) : slice->width;
18193+ return (ptrdiff_t)stbir_info->effective_channels * width;
18194+}
18195+
18196+static void stbir__resample_vertical_gather(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n, int contrib_n0, int contrib_n1, float const * vertical_coefficients, stbir__column_slice const * slice )
18197+{
18198+ float* encode_buffer = split_info->vertical_buffer;
18199+ float* decode_buffer = split_info->decode_buffer;
18200+ int vertical_first = stbir_info->vertical_first;
18201+ ptrdiff_t width_times_channels = stbir__vertical_width_times_channels( stbir_info, slice );
18202+
18203+ STBIR_ASSERT( stbir_info->vertical.is_gather );
18204+
18205+ // loop over the contributing scanlines and scale into the buffer
18206+ STBIR_PROFILE_START( vertical );
18207+ {
18208+ int k = 0, total = contrib_n1 - contrib_n0 + 1;
18209+ STBIR_ASSERT( total > 0 );
18210+ do {
18211+ float const * inputs[8];
18212+ int i, cnt = total;
18213+ if ( cnt > 8 )
18214+ cnt = 8;
18215+ for( i = 0 ; i < cnt ; i++ )
18216+ inputs[ i ] = stbir__get_ring_buffer_scanline(stbir_info, split_info, k+i+contrib_n0 );
18217+
18218+ // call the N scanlines at a time function (up to 8 scanlines of blending at once)
18219+ ((k==0)?stbir__vertical_gathers:stbir__vertical_gathers_continues)[cnt-1]( (vertical_first) ? decode_buffer : encode_buffer, vertical_coefficients + k, inputs, inputs[0] + width_times_channels );
18220+ k += cnt;
18221+ total -= cnt;
18222+ } while ( total );
18223+ }
18224+ STBIR_PROFILE_END( vertical );
18225+
18226+ if ( vertical_first )
18227+ {
18228+ // Now resample the gathered vertical data in the horizontal axis into the encode buffer
18229+ decode_buffer[ width_times_channels ] = 0.0f; // clear two over for horizontals with a remnant of 3
18230+ decode_buffer[ width_times_channels+1 ] = 0.0f;
18231+ stbir__resample_horizontal_gather(stbir_info, encode_buffer, decode_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
18232+ }
18233+
18234+ stbir__encode_scanline( stbir_info, encode_buffer, n, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
18235+}
18236+
18237+static void stbir__decode_and_resample_for_vertical_gather_loop(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n, stbir__column_slice const * slice)
18238+{
18239+ int ring_buffer_index;
18240+ float* ring_buffer;
18241+
18242+ // Decode the nth scanline from the source image into the decode buffer.
18243+ stbir__decode_scanline( stbir_info, n, split_info->decode_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
18244+
18245+ // update new end scanline
18246+ split_info->ring_buffer_last_scanline = n;
18247+
18248+ // get ring buffer
18249+ ring_buffer_index = (split_info->ring_buffer_begin_index + (split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline)) % stbir_info->ring_buffer_num_entries;
18250+ ring_buffer = stbir__get_ring_buffer_entry(stbir_info, split_info, ring_buffer_index);
18251+
18252+ // Now resample it into the ring buffer.
18253+ stbir__resample_horizontal_gather( stbir_info, ring_buffer, split_info->decode_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
18254+
18255+ // Now it's sitting in the ring buffer ready to be used as source for the vertical sampling.
18256+}
18257+
18258+// A trimmed zero coefficient can make a row's range start later than the next row's ((4,4) then (3,6)), and the ring only
18259+// loads forward. So look ahead up to max_look rows from y for the lowest first scanline; *out_row is which row had it.
18260+static int stbir__lowest_scanline_needed( stbir__sampler const * vertical, int y, int max_look, int * out_row )
18261+{
18262+ int start_n0, lowest_n0, n0, n1, j;
18263+
18264+ stbir__output_range( vertical, y, &start_n0, &n1 );
18265+ lowest_n0 = start_n0;
18266+ *out_row = 0;
18267+ for (j = 1; j <= max_look; j++)
18268+ {
18269+ stbir__output_range( vertical, y + j, &n0, &n1 );
18270+ if ( n0 > start_n0 )
18271+ break;
18272+ if ( n0 < lowest_n0 )
18273+ {
18274+ lowest_n0 = n0;
18275+ *out_row = j;
18276+ }
18277+ }
18278+ return lowest_n0;
18279+}
18280+
18281+static void stbir__vertical_gather_loop( stbir__info const * stbir_info, stbir__per_split_info* split_info, int split_count, stbir__column_slice const * slice )
18282+{
18283+ stbir__sampler const * vertical = &stbir_info->vertical;
18284+ int y, start_output_y, end_output_y;
18285+
18286+ STBIR_ASSERT( vertical->is_gather );
18287+
18288+ start_output_y = split_info->start_output_y;
18289+ end_output_y = split_info[split_count-1].end_output_y;
18290+
18291+ // initialize the ring buffer for gathering: start it at the lowest scanline the first rows need
18292+ {
18293+ int look = vertical->filter_pixel_margin * 3, unused;
18294+ if ( look > ( end_output_y - start_output_y - 1 ) )
18295+ look = end_output_y - start_output_y - 1;
18296+ split_info->ring_buffer_begin_index = 0;
18297+ split_info->ring_buffer_first_scanline = stbir__lowest_scanline_needed( vertical, start_output_y, look, &unused );
18298+ split_info->ring_buffer_last_scanline = split_info->ring_buffer_first_scanline - 1; // means "empty"
18299+ }
18300+
18301+ for (y = start_output_y; y < end_output_y; y++)
18302+ {
18303+ int in_first_scanline, in_last_scanline;
18304+ float const * vertical_coefficients = stbir__output_coefficients( vertical, y );
18305+
18306+ stbir__output_range( vertical, y, &in_first_scanline, &in_last_scanline );
18307+
18308+ // rows with a zero weight at either end aren't worth decoding (the pack's shifted sets when the vertical shares the
18309+ // horizontal's tables, kernels that are exactly zero at their support's edge)
18310+ while ( ( in_first_scanline < in_last_scanline ) && ( vertical_coefficients[0] == 0.0f ) )
18311+ {
18312+ ++in_first_scanline;
18313+ ++vertical_coefficients;
18314+ }
18315+ while ( ( in_last_scanline > in_first_scanline ) && ( vertical_coefficients[ in_last_scanline - in_first_scanline ] == 0.0f ) )
18316+ --in_last_scanline;
18317+
18318+ // make sure the indexing hasn't broken
18319+ STBIR_ASSERT( in_first_scanline >= split_info->ring_buffer_first_scanline );
18320+
18321+ // Load in new scanlines
18322+ while (in_last_scanline > split_info->ring_buffer_last_scanline)
18323+ {
18324+ STBIR_ASSERT( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) <= stbir_info->ring_buffer_num_entries );
18325+
18326+ // make sure there was room in the ring buffer when we add new scanlines
18327+ if ( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) == stbir_info->ring_buffer_num_entries )
18328+ {
18329+ split_info->ring_buffer_first_scanline++;
18330+ split_info->ring_buffer_begin_index++;
18331+ }
18332+
18333+ if ( stbir_info->vertical_first )
18334+ {
18335+ float * ring_buffer = stbir__get_ring_buffer_scanline( stbir_info, split_info, ++split_info->ring_buffer_last_scanline );
18336+ // Decode the nth scanline from the source image into the decode buffer.
18337+ stbir__decode_scanline( stbir_info, split_info->ring_buffer_last_scanline, ring_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
18338+ }
18339+ else
18340+ {
18341+ stbir__decode_and_resample_for_vertical_gather_loop(stbir_info, split_info, split_info->ring_buffer_last_scanline + 1, slice);
18342+ }
18343+ }
18344+
18345+ // Now all buffers should be ready to write a row of vertical sampling, so do it.
18346+ stbir__resample_vertical_gather(stbir_info, split_info, y, in_first_scanline, in_last_scanline, vertical_coefficients, slice );
18347+ }
18348+}
18349+
18350+#define STBIR__FLOAT_EMPTY_MARKER 3.0e+38F
18351+#define STBIR__FLOAT_BUFFER_IS_EMPTY(ptr) ((ptr)[0]==STBIR__FLOAT_EMPTY_MARKER)
18352+
18353+static void stbir__encode_first_scanline_from_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info, stbir__column_slice const * slice)
18354+{
18355+ // evict a scanline out into the output buffer
18356+ float* ring_buffer_entry = stbir__get_ring_buffer_entry(stbir_info, split_info, split_info->ring_buffer_begin_index );
18357+
18358+ // dump the scanline out
18359+ stbir__encode_scanline( stbir_info, ring_buffer_entry, split_info->ring_buffer_first_scanline, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
18360+
18361+ // mark it as empty
18362+ ring_buffer_entry[ 0 ] = STBIR__FLOAT_EMPTY_MARKER;
18363+
18364+ // advance the first scanline
18365+ split_info->ring_buffer_first_scanline++;
18366+ if ( ++split_info->ring_buffer_begin_index == stbir_info->ring_buffer_num_entries )
18367+ split_info->ring_buffer_begin_index = 0;
18368+}
18369+
18370+static void stbir__horizontal_resample_and_encode_first_scanline_from_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info, stbir__column_slice const * slice)
18371+{
18372+ // evict a scanline out into the output buffer
18373+
18374+ float* ring_buffer_entry = stbir__get_ring_buffer_entry(stbir_info, split_info, split_info->ring_buffer_begin_index );
18375+
18376+ // Now resample it into the buffer.
18377+ stbir__resample_horizontal_gather( stbir_info, split_info->vertical_buffer, ring_buffer_entry, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
18378+
18379+ // dump the scanline out
18380+ stbir__encode_scanline( stbir_info, split_info->vertical_buffer, split_info->ring_buffer_first_scanline, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
18381+
18382+ // mark it as empty
18383+ ring_buffer_entry[ 0 ] = STBIR__FLOAT_EMPTY_MARKER;
18384+
18385+ // advance the first scanline
18386+ split_info->ring_buffer_first_scanline++;
18387+ if ( ++split_info->ring_buffer_begin_index == stbir_info->ring_buffer_num_entries )
18388+ split_info->ring_buffer_begin_index = 0;
18389+}
18390+
18391+static void stbir__resample_vertical_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n0, int n1, float const * vertical_coefficients, float const * vertical_buffer, float const * vertical_buffer_end )
18392+{
18393+ STBIR_ASSERT( !stbir_info->vertical.is_gather );
18394+
18395+ STBIR_PROFILE_START( vertical );
18396+ {
18397+ int k = 0, total = n1 - n0 + 1;
18398+ STBIR_ASSERT( total > 0 );
18399+ do {
18400+ float * outputs[8];
18401+ int i, n = total; if ( n > 8 ) n = 8;
18402+ for( i = 0 ; i < n ; i++ )
18403+ {
18404+ outputs[ i ] = stbir__get_ring_buffer_scanline(stbir_info, split_info, k+i+n0 );
18405+ if ( ( i ) && ( STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[i] ) != STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[0] ) ) ) // make sure runs are of the same type
18406+ {
18407+ n = i;
18408+ break;
18409+ }
18410+ }
18411+ // call the scatter to N scanlines at a time function (up to 8 scanlines of scattering at once)
18412+ ((STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[0] ))?stbir__vertical_scatter_sets:stbir__vertical_scatter_blends)[n-1]( outputs, vertical_coefficients + k, vertical_buffer, vertical_buffer_end );
18413+ k += n;
18414+ total -= n;
18415+ } while ( total );
18416+ }
18417+
18418+ STBIR_PROFILE_END( vertical );
18419+}
18420+
18421+typedef void stbir__handle_scanline_for_scatter_func(stbir__info const * stbir_info, stbir__per_split_info* split_info, stbir__column_slice const * slice);
18422+
18423+static void stbir__vertical_scatter_loop( stbir__info const * stbir_info, stbir__per_split_info* split_info, int split_count, stbir__column_slice const * slice )
18424+{
18425+ int y, start_output_y, end_output_y, start_input_y, end_input_y;
18426+ stbir__contributors* vertical_contributors = stbir_info->vertical.contributors;
18427+ float const * vertical_coefficients = stbir_info->vertical.coefficients;
18428+ stbir__handle_scanline_for_scatter_func * handle_scanline_for_scatter;
18429+ void * scanline_scatter_buffer;
18430+ void * scanline_scatter_buffer_end;
18431+ int on_first_input_y, last_input_y;
18432+ ptrdiff_t width_times_channels = stbir__vertical_width_times_channels( stbir_info, slice );
18433+
18434+ STBIR_ASSERT( !stbir_info->vertical.is_gather );
18435+
18436+ start_output_y = split_info->start_output_y;
18437+ end_output_y = split_info[split_count-1].end_output_y; // may do multiple split counts
18438+
18439+ start_input_y = split_info->start_input_y;
18440+ end_input_y = split_info[split_count-1].end_input_y;
18441+
18442+ // adjust for starting offset start_input_y
18443+ y = start_input_y + stbir_info->vertical.filter_pixel_margin;
18444+ vertical_contributors += y ;
18445+ vertical_coefficients += (size_t)stbir_info->vertical.coefficient_width * y;
18446+
18447+ if ( stbir_info->vertical_first )
18448+ {
18449+ handle_scanline_for_scatter = stbir__horizontal_resample_and_encode_first_scanline_from_scatter;
18450+ scanline_scatter_buffer = split_info->decode_buffer;
18451+ }
18452+ else
18453+ {
18454+ handle_scanline_for_scatter = stbir__encode_first_scanline_from_scatter;
18455+ scanline_scatter_buffer = split_info->vertical_buffer;
18456+ }
18457+ scanline_scatter_buffer_end = ( (char*) scanline_scatter_buffer ) + sizeof( float ) * width_times_channels;
18458+
18459+ // initialize the ring buffer for scattering
18460+ split_info->ring_buffer_first_scanline = start_output_y;
18461+ split_info->ring_buffer_last_scanline = -1;
18462+ split_info->ring_buffer_begin_index = -1;
18463+
18464+ // mark all the buffers as empty to start
18465+ for( y = 0 ; y < stbir_info->ring_buffer_num_entries ; y++ )
18466+ {
18467+ float * decode_buffer = stbir__get_ring_buffer_entry( stbir_info, split_info, y );
18468+ decode_buffer[ width_times_channels ] = 0.0f; // clear two over for horizontals with a remnant of 3
18469+ decode_buffer[ width_times_channels+1 ] = 0.0f;
18470+ decode_buffer[0] = STBIR__FLOAT_EMPTY_MARKER; // only used on scatter
18471+ }
18472+
18473+ // do the loop in input space
18474+ on_first_input_y = 1;
18475+ last_input_y = start_input_y;
18476+ for (y = start_input_y ; y < end_input_y; y++)
18477+ {
18478+ int out_first_scanline, out_last_scanline;
18479+
18480+ out_first_scanline = vertical_contributors->n0;
18481+ out_last_scanline = vertical_contributors->n1;
18482+
18483+ // this input row is ours if the output rows it contributes to overlap the split's rows at all
18484+ if ( ( out_last_scanline >= out_first_scanline ) && ( out_last_scanline >= start_output_y ) && ( out_first_scanline < end_output_y ) )
18485+ {
18486+ float const * coefficients = vertical_coefficients;
18487+
18488+ // keep track of the range actually seen for the next resize
18489+ last_input_y = y;
18490+ if ( ( on_first_input_y ) && ( y > start_input_y ) )
18491+ split_info->start_input_y = y;
18492+ on_first_input_y = 0;
18493+
18494+ // clip the region
18495+ if ( out_first_scanline < start_output_y )
18496+ {
18497+ coefficients += start_output_y - out_first_scanline;
18498+ out_first_scanline = start_output_y;
18499+ }
18500+
18501+ if ( out_last_scanline >= end_output_y )
18502+ out_last_scanline = end_output_y - 1;
18503+
18504+ STBIR_ASSERT( out_last_scanline - out_first_scanline + 1 <= stbir_info->ring_buffer_num_entries ); // (the ring is capped to the band's rows, so after the clip)
18505+
18506+ // if very first scanline, init the index
18507+ if (split_info->ring_buffer_begin_index < 0)
18508+ split_info->ring_buffer_begin_index = out_first_scanline - start_output_y;
18509+
18510+ STBIR_ASSERT( split_info->ring_buffer_begin_index <= out_first_scanline );
18511+
18512+ // Decode the nth scanline from the source image into the decode buffer.
18513+ stbir__decode_scanline( stbir_info, y, split_info->decode_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
18514+
18515+ // When horizontal first, we resample horizontally into the vertical buffer before we scatter it out
18516+ if ( !stbir_info->vertical_first )
18517+ stbir__resample_horizontal_gather( stbir_info, split_info->vertical_buffer, split_info->decode_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
18518+
18519+ // Now it's sitting in the buffer ready to be distributed into the ring buffers.
18520+
18521+ // evict from the ring buffer if it is full and this row needs a new entry
18522+ if ( ( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) == stbir_info->ring_buffer_num_entries ) &&
18523+ ( out_last_scanline > split_info->ring_buffer_last_scanline ) )
18524+ handle_scanline_for_scatter( stbir_info, split_info, slice );
18525+
18526+ // Now the horizontal buffer is ready to write to all ring buffer rows, so do it.
18527+ stbir__resample_vertical_scatter(stbir_info, split_info, out_first_scanline, out_last_scanline, coefficients, (float*)scanline_scatter_buffer, (float*)scanline_scatter_buffer_end );
18528+
18529+ // update the end of the buffer
18530+ if ( out_last_scanline > split_info->ring_buffer_last_scanline )
18531+ split_info->ring_buffer_last_scanline = out_last_scanline;
18532+ }
18533+ ++vertical_contributors;
18534+ vertical_coefficients += stbir_info->vertical.coefficient_width;
18535+ }
18536+
18537+ // now evict the scanlines that are left over in the ring buffer
18538+ while ( split_info->ring_buffer_first_scanline < end_output_y )
18539+ handle_scanline_for_scatter(stbir_info, split_info, slice);
18540+
18541+ // update the end_input_y if we do multiple resizes with the same data
18542+ ++last_input_y;
18543+ for( y = 0 ; y < split_count; y++ )
18544+ if ( split_info[y].end_input_y > last_input_y )
18545+ split_info[y].end_input_y = last_input_y;
18546+}
18547+
18548+
18549+static void stbir__set_sampler(stbir__sampler * samp, stbir_filter filter, stbir__kernel_callback_new * kernel_new, stbir__support_callback_new * support_new, stbir__radius_callback_new * radius, stbir__kernel_callback * kernel_old, stbir__support_callback * support_old, stbir_edge edge, stbir__scale_info * scale_info, int always_gather, void * user_data )
18550+{
18551+ // set filter
18552+ if (filter == 0)
18553+ {
18554+ filter = STBIR_DEFAULT_FILTER_DOWNSAMPLE; // default to downsample
18555+ if ( stbir__scale_is_upsample( scale_info ) )
18556+ {
18557+ if ( stbir__scale_is_copy( scale_info ) )
18558+ filter = STBIR_FILTER_POINT_SAMPLE;
18559+ else
18560+ filter = STBIR_DEFAULT_FILTER_UPSAMPLE;
18561+ }
18562+ }
18563+ samp->filter_enum = filter;
18564+
18565+ STBIR_ASSERT(samp->filter_enum != 0);
18566+ STBIR_ASSERT((unsigned)samp->filter_enum < STBIR_FILTER_OTHER);
18567+ samp->filter_kernel_new = stbir__builtin_kernels[ filter ];
18568+ samp->filter_kernel_old = 0;
18569+ samp->filter_support_new = stbir__builtin_supports[ filter ];
18570+ samp->filter_radius = stbir__builtin_radii[ filter ];
18571+ samp->filter_support_old = 0;
18572+
18573+ // a user filter is a kernel with a support or a radius (stbir_set_filter_callbacks_new refuses both), or an old
18574+ // float kernel with an old float support (stbir_set_filter_callbacks)
18575+ if ( ( kernel_new && ( support_new || radius ) ) || ( kernel_old && support_old ) )
18576+ {
18577+ samp->filter_kernel_new = kernel_new;
18578+ samp->filter_kernel_old = kernel_old;
18579+ samp->filter_support_new = support_new;
18580+ samp->filter_radius = radius;
18581+ samp->filter_support_old = support_old;
18582+ samp->filter_enum = STBIR_FILTER_OTHER;
18583+ }
18584+
18585+ samp->edge = edge;
18586+ samp->filter_pixel_width = stbir__get_filter_pixel_width( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, scale_info, user_data );
18587+ // Gather is always better, but in extreme downsamples, you have to most or all of the data in memory
18588+ // For horizontal, we always have all the pixels, so we always use gather here (always_gather==1).
18589+ // For vertical, we use gather if scaling up (which means we will have samp->filter_pixel_width
18590+ // scanlines in memory at once).
18591+ samp->is_gather = 0;
18592+ if ( stbir__scale_is_upsample( scale_info ) )
18593+ samp->is_gather = 1;
18594+ else if ( ( always_gather ) || ( samp->filter_pixel_width <= STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT ) )
18595+ samp->is_gather = 2;
18596+
18597+ // pre calculate stuff based on the above
18598+ samp->coefficient_width = stbir__get_coefficient_width(samp, samp->is_gather, user_data);
18599+
18600+ // filter_pixel_width is the conservative size in pixels of input that affect an output pixel.
18601+ // In rare cases (only with 2 pix to 1 pix with the default filters), it's possible that the
18602+ // filter will extend before or after the scanline beyond just one extra entire copy of the
18603+ // scanline (we would hit the edge twice). We don't let you do that, so we clamp the total
18604+ // width to 3x the total of input pixel (once for the scanline, once for the left side
18605+ // overhang, and once for the right side). We only do this for edge mode, since the other
18606+ // modes can just re-edge clamp back in again.
18607+ if ( edge == STBIR_EDGE_WRAP )
18608+ if ( ( samp->filter_pixel_width / 3 ) > scale_info->input_full_size )
18609+ samp->filter_pixel_width = scale_info->input_full_size * 3;
18610+
18611+ // This is how much to expand buffers to account for filters seeking outside
18612+ // the image boundaries.
18613+ samp->filter_pixel_margin = samp->filter_pixel_width / 2;
18614+
18615+ // filter_pixel_margin is the amount that this filter can overhang on just one side of either
18616+ // end of the scanline (left or the right). Since we only allow you to overhang 1 scanline's
18617+ // worth of pixels, we clamp this one side of overhang to the input scanline size. Again,
18618+ // this clamping only happens in rare cases with the default filters (2 pix to 1 pix).
18619+ if ( edge == STBIR_EDGE_WRAP )
18620+ if ( samp->filter_pixel_margin > scale_info->input_full_size )
18621+ samp->filter_pixel_margin = scale_info->input_full_size;
18622+
18623+ samp->num_contributors = stbir__get_contributors(samp, samp->is_gather);
18624+
18625+ samp->entry_floats = 0;
18626+ samp->shared_period = 0;
18627+ samp->shared_prologue = 0;
18628+ samp->shared_epilogue = 0;
18629+ samp->contributors_size = (size_t)samp->num_contributors * sizeof(stbir__contributors);
18630+ samp->coefficients_size = (size_t)samp->num_contributors * STBIR__COLUMN_ENTRY_FLOATS( samp->coefficient_width ) * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // room for the horizontal's column entry ints, plus padding
18631+}
18632+
18633+// Walks input pixels from start toward end (step is -1 or 1) and returns the last one that contributes to any output,
18634+// or fallback if none did. When the output region is only a pixel or two wide the starting guess can be past the
18635+// whole region, so a non contributing pixel only stops the walk once contributing ones have been seen.
18636+static int stbir__last_contributing_input( int start, int end, int step, int fallback, stbir_real in_pixels_radius, stbir_real support, stbir__scale_info const * scale_info )
18637+{
18638+ int n = start;
18639+ int last = fallback;
18640+ int seen = 0;
18641+
18642+ while ( ( step < 0 ) ? ( n >= end ) : ( n <= end ) )
18643+ {
18644+ int out_first_pixel, out_last_pixel;
18645+ stbir__input_out_pixel_range( scale_info, n, in_pixels_radius, support, &out_first_pixel, &out_last_pixel );
18646+ if ( out_first_pixel > out_last_pixel )
18647+ {
18648+ if ( seen )
18649+ break;
18650+ }
18651+ else
18652+ {
18653+ last = n;
18654+ seen = 1;
18655+ }
18656+ n += step;
18657+ }
18658+ return last;
18659+}
18660+
18661+static void stbir__get_conservative_extents( stbir__sampler * samp, stbir__contributors * range, void * user_data )
18662+{
18663+ stbir_real scale = samp->scale_info.scale;
18664+ stbir_real in_pixels_radius = stbir__filter_radius_in_input_pixels( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, &samp->scale_info, user_data );
18665+ int input_full_size = samp->scale_info.input_full_size;
18666+ stbir_edge edge = samp->edge;
18667+ stbir_real inv_scale = samp->scale_info.inv_scale;
18668+
18669+ STBIR_ASSERT( samp->is_gather != 0 );
18670+
18671+ if ( samp->is_gather == 1 )
18672+ {
18673+ int in_first_pixel, in_last_pixel;
18674+ stbir_real filter_support = in_pixels_radius;
18675+ stbir_real out_filter_radius = filter_support * scale;
18676+
18677+ stbir__output_in_pixel_range( &samp->scale_info, 0, out_filter_radius, filter_support, edge, &in_first_pixel, &in_last_pixel );
18678+ range->n0 = in_first_pixel;
18679+ stbir__output_in_pixel_range( &samp->scale_info, samp->scale_info.output_sub_size - 1, out_filter_radius, filter_support, edge, &in_first_pixel, &in_last_pixel );
18680+ range->n1 = in_last_pixel;
18681+ }
18682+ else if ( samp->is_gather == 2 ) // downsample gather, refine
18683+ {
18684+ stbir_real filter_support = in_pixels_radius / inv_scale; // in output pixels
18685+ int filter_pixel_margin = samp->filter_pixel_margin;
18686+ int output_sub_size = samp->scale_info.output_sub_size;
18687+ int in_first_pixel, in_last_pixel;
18688+
18689+ // get a conservative area of the input range
18690+ stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, (stbir_real)0.0, (stbir_real)0.0, samp->scale_info.inv_scale, samp->scale_info.pixel_shift, input_full_size, edge );
18691+ range->n0 = in_first_pixel;
18692+ stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, (stbir_real)output_sub_size, (stbir_real)0.0, samp->scale_info.inv_scale, samp->scale_info.pixel_shift, input_full_size, edge );
18693+ range->n1 = in_last_pixel;
18694+
18695+ // walk down through the margin from the start of the area to find the bottom, and up from its end to find the top
18696+ // (the coefficient generator never goes past input_full_size + filter_pixel_margin - 1)
18697+ range->n0 = stbir__last_contributing_input( range->n0 + 1, -filter_pixel_margin, -1, range->n0, in_pixels_radius, filter_support, &samp->scale_info );
18698+ range->n1 = stbir__last_contributing_input( range->n1 - 1, input_full_size + filter_pixel_margin - 1, 1, range->n1, in_pixels_radius, filter_support, &samp->scale_info );
18699+ }
18700+
18701+ // when polyphase, the coefficient generator only calculates the first numerator outputs and then copies them
18702+ // forward shifted by denominator (exact integer math). the top of the range above was calculated directly at the
18703+ // far end of the image in floating point, which can round the other way, so widen by one - and when the fraction is
18704+ // an approximation of the scale, the copies drift from the real positions by up to approximated_drift input pixels.
18705+ if ( ( samp->scale_info.scale_is_rational ) && ( (int)samp->scale_info.scale_numerator < samp->scale_info.output_sub_size ) )
18706+ range->n1 += 1 + samp->scale_info.approximated_drift;
18707+
18708+ // with WRAP the range is allowed to hang off either edge (those pixels get decoded from the other end of
18709+ // the scanline), the other modes fold everything back into the scanline so clamp them to it
18710+ if ( samp->edge != STBIR_EDGE_WRAP )
18711+ {
18712+ // for reflect, the margin taps get folded back into the image, and on tiny inputs we
18713+ // land beyond, so widen to include them
18714+ if ( samp->edge == STBIR_EDGE_REFLECT )
18715+ {
18716+ if ( range->n0 < 0 )
18717+ range->n1 = stbir__max( range->n1, stbir__min( -range->n0, input_full_size - 1 ) );
18718+ if ( range->n1 >= input_full_size )
18719+ range->n0 = stbir__min( range->n0, stbir__max( 0, ( input_full_size - 1 ) - ( range->n1 - input_full_size ) ) ); // ( 2 x size - n1 - 1, written so it can't overflow at 2^30 )
18720+ }
18721+
18722+ // for non-edge-wrap modes, we never read over the edge, so clamp
18723+ if ( range->n0 < 0 )
18724+ range->n0 = 0;
18725+ if ( range->n1 >= input_full_size )
18726+ range->n1 = input_full_size - 1;
18727+ }
18728+}
18729+
18730+static void stbir__get_split_info( stbir__per_split_info* split_info, int splits, int output_height, int vertical_pixel_margin, int input_full_height, int is_gather, stbir__sampler const * vertical )
18731+{
18732+ int start_y = 0;
18733+ int remaining = output_height;
18734+ int i;
18735+
18736+ for (i = 0; i < splits; i++)
18737+ {
18738+ int rows = remaining / ( splits - i );
18739+
18740+ split_info[i].start_output_y = start_y;
18741+ split_info[i].end_output_y = start_y + rows;
18742+
18743+ // Don't start a split on a row whose range starts later than a following row's (see stbir__lowest_scanline_needed):
18744+ // move the rows up to that one into the previous split, so it ends with the unusual row instead of this one
18745+ // starting on it. Only this split's rows are looked at (the next split's first row is its own problem).
18746+ if ( ( is_gather ) && ( i ) && ( rows > 1 ) )
18747+ {
18748+ int look = vertical_pixel_margin * 3;
18749+ int move;
18750+
18751+ if ( look > ( rows - 1 ) )
18752+ look = ( rows - 1 );
18753+ stbir__lowest_scanline_needed( vertical, start_y, look, &move );
18754+ split_info[i-1].end_output_y += move;
18755+ split_info[i].start_output_y += move;
18756+ }
18757+
18758+ start_y += rows;
18759+ remaining -= rows;
18760+
18761+ // scatter range (updated to minimum as you run it)
18762+ split_info[i].start_input_y = -vertical_pixel_margin;
18763+ split_info[i].end_input_y = input_full_height + vertical_pixel_margin;
18764+ }
18765+}
18766+
18767+static void stbir__free_internal_mem( stbir__info *info )
18768+{
18769+ #define STBIR__FREE_AND_CLEAR( ptr ) { if ( ptr ) { void * p = (ptr); (ptr) = 0; STBIR_FREE( p, info->user_data); } }
18770+
18771+ if ( info )
18772+ {
18773+ #ifndef STBIR__SEPARATE_ALLOCATIONS
18774+ STBIR__FREE_AND_CLEAR( info->alloced_mem );
18775+ #else
18776+ // a build whose allocation failed part way comes through here too: info and its pointer arrays are zeroed when
18777+ // allocated, so every pointer is either an allocation or null
18778+ int i,j;
18779+
18780+ if ( info->split_info )
18781+ {
18782+ for( i = 0 ; i < info->splits ; i++ )
18783+ {
18784+ stbir__per_split_info * split = info->split_info + i;
18785+
18786+ if ( split->ring_buffers )
18787+ {
18788+ for( j = 0 ; j < info->alloc_ring_buffer_num_entries ; j++ )
18789+ {
18790+ float * ring_buffer = split->ring_buffers[j];
18791+ if ( ring_buffer )
18792+ {
18793+ #ifdef STBIR_SIMD8
18794+ if ( info->effective_channels == 3 )
18795+ --ring_buffer; // avx in 3 channel mode needs one float at the start of the buffer
18796+ #endif
18797+ STBIR_FREE( ring_buffer, info->user_data );
18798+ }
18799+ }
18800+ STBIR__FREE_AND_CLEAR( split->ring_buffers );
18801+ }
18802+
18803+ if ( split->decode_buffer_alloc )
18804+ {
18805+ #ifdef STBIR_SIMD8
18806+ if ( info->effective_channels == 3 )
18807+ split->decode_buffer_alloc = ( (float*)split->decode_buffer_alloc ) - 1; // avx in 3 channel mode needs one float at the start of the buffer
18808+ #endif
18809+ STBIR__FREE_AND_CLEAR( split->decode_buffer_alloc );
18810+ }
18811+ STBIR__FREE_AND_CLEAR( split->vertical_buffer_alloc );
18812+ }
18813+ STBIR__FREE_AND_CLEAR( info->split_info );
18814+ }
18815+ STBIR__FREE_AND_CLEAR( info->column_slices );
18816+ if ( info->vertical.contributors != info->horizontal.contributors ) // (when shared, the vertical's coefficients point two ints into the horizontal's)
18817+ {
18818+ STBIR__FREE_AND_CLEAR( info->vertical.coefficients );
18819+ STBIR__FREE_AND_CLEAR( info->vertical.contributors );
18820+ }
18821+ STBIR__FREE_AND_CLEAR( info->horizontal.coefficients );
18822+ STBIR__FREE_AND_CLEAR( info->horizontal.contributors );
18823+ STBIR__FREE_AND_CLEAR( info->alloced_mem );
18824+ STBIR_FREE( info, info->user_data );
18825+ #endif
18826+ }
18827+
18828+ #undef STBIR__FREE_AND_CLEAR
18829+}
18830+
18831+static int stbir__get_max_split( int splits, int height )
18832+{
18833+ int i;
18834+ int max = 0;
18835+
18836+ for( i = 0 ; i < splits ; i++ )
18837+ {
18838+ int each = height / ( splits - i );
18839+ if ( each > max )
18840+ max = each;
18841+ height -= each;
18842+ }
18843+ return max;
18844+}
18845+
18846+static stbir__horizontal_gather_channels_func ** stbir__horizontal_gather_n_coeffs_funcs[8] =
18847+{
18848+ 0, stbir__horizontal_gather_1_channels_with_n_coeffs_funcs, stbir__horizontal_gather_2_channels_with_n_coeffs_funcs, stbir__horizontal_gather_3_channels_with_n_coeffs_funcs, stbir__horizontal_gather_4_channels_with_n_coeffs_funcs, 0,0, stbir__horizontal_gather_7_channels_with_n_coeffs_funcs
18849+};
18850+
18851+static stbir__horizontal_gather_channels_func ** stbir__horizontal_gather_channels_funcs[8] =
18852+{
18853+ 0, stbir__horizontal_gather_1_channels_funcs, stbir__horizontal_gather_2_channels_funcs, stbir__horizontal_gather_3_channels_funcs, stbir__horizontal_gather_4_channels_funcs, 0,0, stbir__horizontal_gather_7_channels_funcs
18854+};
18855+
18856+// the polyphase period 1 kernels, [channels][coefficient count] (see the horizontal kernel section)
18857+static stbir__horizontal_gather_channels_func ** stbir__horizontal_gather_period1_funcs[8] =
18858+{
18859+ 0, stbir__horizontal_gather_1_channels_period1_funcs, stbir__horizontal_gather_2_channels_period1_funcs, stbir__horizontal_gather_3_channels_period1_funcs, stbir__horizontal_gather_4_channels_period1_funcs, 0,0, stbir__horizontal_gather_7_channels_period1_funcs
18860+};
18861+
18862+// debug overrides (the tests set these; they are statics, so per translation unit): 0 = off
18863+static int stbir__force_vertical_first = 0; // 1 = horizontal first, 2 = vertical first (dotimings, refcheck -vfirst)
18864+static int stbir__chose_vertical_first = 0; // (read only) what the cost model picked for the last build
18865+static int stbir__split_mode = 0; // 1 = row bands only, 2 = column groups only (refcheck_threads -row_split / -colsplit)
18866+static int stbir__split_force_cols = 0; // force this many column groups, the bands are splits / this (refcheck_threads -grid)
18867+static int stbir__column_slices = 0; // force this many column slices instead of the L2 fit (refcheck_time cfg N:)
18868+static int stbir__4K_vertical_dest_shift = 0; // bytes added to the vertical gather destination's page offset (refcheck_time -vshift)
18869+
18870+// Vertical first or horizontal first? It matters most when the two axes scale very differently (2x wider and half the
18871+// height: the vertical first order is around 3x faster), and 20-40% in ordinary cases. The decision is a kernel cost
18872+// model: both orders move the same decoded pixels, so the question is only how much each kernel charges per byte it
18873+// streams and the multiplies each direction does (vertical is more SIMDable than horizontal). We calculate the costs as:
18874+//
18875+// vertical first: (taps_v + 1) / s_h * rv + (1/s_h + 1) * rh (vertical over input-width rows, horizontal over output rows)
18876+// horizontal first: (1/s_v) * (1/s_h + 1) * rh + (taps_v + 1) * rv (horizontal over every input row, vertical over output-width rows)
18877+// (1/s = input/output per axis, with WRAP's margin pixels and scanlines added to the input)
18878+//
18879+// rh and rv are the scale factors for the horizontal and vertical costs. We then measure a bunch of horizontal
18880+// first and vertical first resizes on a bunch of machines to train small tables to give us a relative speed of
18881+// each direction.
18882+
18883+// horizontal tap count -> table column: 1, 2, 3, 4, 5-6, 7-8, 9-12, 13-17, 18+
18884+static int stbir__horizontal_cost_bucket( int taps )
18885+{
18886+ if ( taps <= 4 ) return ( taps < 1 ) ? 0 : taps - 1;
18887+ if ( taps <= 6 ) return 4;
18888+ if ( taps <= 8 ) return 5;
18889+ if ( taps <= 12 ) return 6;
18890+ if ( taps <= 17 ) return 7;
18891+ return 8;
18892+}
18893+
18894+// [1,2,3,4,7 channels][bucket]: horizontal kernel cost per byte streamed, relative to the vertical kernel's
18895+static float stbir__horizontal_cost[5][9] =
18896+{
18897+ { 12.472f, 6.718f, 6.656f, 8.008f, 5.957f, 6.068f, 6.474f, 7.646f, 10.184f },
18898+ { 8.067f, 4.192f, 5.091f, 4.906f, 4.816f, 5.584f, 5.741f, 6.971f, 6.182f },
18899+ { 5.495f, 3.718f, 3.206f, 7.861f, 4.816f, 5.584f, 5.741f, 5.584f, 6.068f },
18900+ { 3.651f, 2.135f, 2.791f, 3.583f, 2.950f, 3.583f, 4.154f, 4.192f, 4.861f },
18901+ { 2.434f, 2.001f, 2.521f, 3.583f, 2.817f, 3.236f, 3.550f, 3.822f, 4.271f },
18902+};
18903+
18904+// the same when the horizontal is an integer downsample (the polyphase period 1 kernels)
18905+static float stbir__horizontal_cost_period1[5][9] =
18906+{
18907+ { 12.406f, 6.718f, 6.534f, 5.186f, 5.114f, 5.138f, 6.656f, 5.431f, 10.374f },
18908+ { 8.001f, 4.154f, 3.583f, 4.003f, 3.276f, 4.473f, 3.683f, 4.231f, 6.013f },
18909+ { 5.024f, 3.033f, 3.583f, 3.683f, 3.268f, 4.432f, 3.650f, 4.231f, 5.283f },
18910+ { 2.976f, 2.076f, 2.665f, 2.452f, 2.354f, 2.766f, 2.715f, 3.327f, 4.684f },
18911+ { 2.184f, 1.928f, 2.342f, 1.983f, 2.233f, 3.005f, 2.715f, 3.421f, 3.858f },
18912+};
18913+
18914+// a x b as a signed 128 bit value, for the exact cost comparison below
18915+static void stbir__mul_64x64( stbir_int64 a, stbir_int64 b, stbir_int64 * hi, stbir_uint64 * lo )
18916+{
18917+ stbir_uint64 ua = (stbir_uint64) a, ub = (stbir_uint64) b;
18918+ stbir_uint64 a0 = ua & 0xffffffffu, a1 = ua >> 32, b0 = ub & 0xffffffffu, b1 = ub >> 32;
18919+ stbir_uint64 p00 = a0 * b0, p01 = a0 * b1, p10 = a1 * b0, p11 = a1 * b1;
18920+ stbir_uint64 mid = ( p00 >> 32 ) + ( p01 & 0xffffffffu ) + ( p10 & 0xffffffffu );
18921+ stbir_uint64 uhi = p11 + ( p01 >> 32 ) + ( p10 >> 32 ) + ( mid >> 32 );
18922+ *lo = ( mid << 32 ) | ( p00 & 0xffffffffu );
18923+ if ( a < 0 ) uhi -= ub; // the unsigned product of the two's complements, corrected to the signed one
18924+ if ( b < 0 ) uhi -= ua;
18925+ *hi = (stbir_int64) uhi;
18926+}
18927+
18928+// a x b <= c x d, exactly
18929+static int stbir__products_le( stbir_int64 a, stbir_int64 b, stbir_int64 c, stbir_int64 d )
18930+{
18931+ stbir_int64 lhi, rhi;
18932+ stbir_uint64 llo, rlo;
18933+ stbir__mul_64x64( a, b, &lhi, &llo );
18934+ stbir__mul_64x64( c, d, &rhi, &rlo );
18935+ return ( lhi < rhi ) || ( ( lhi == rhi ) && ( llo <= rlo ) );
18936+}
18937+
18938+// WRAP is the one edge mode that changes the amount of work: the decoded rows carry the filter's margin pixels on both
18939+// ends and the vertical runs over margin scanlines, so the input is effectively taps wider / taller.
18940+static int stbir__should_do_vertical_first( int channel_index, int horizontal_filter_pixel_width, int horizontal_input_size, int horizontal_output_size, stbir_edge horizontal_edge, int horizontal_is_period1,
18941+ int vertical_filter_pixel_width, int vertical_input_size, int vertical_output_size, stbir_edge vertical_edge )
18942+{
18943+ stbir_int64 horizontal_input, vertical_input, horizontal_output, vertical_output, vertical_taps, cost_ratio;
18944+ int bucket, vertical_first;
18945+
18946+ // the horizontal kernel's cost relative to the vertical's, in 1/1000ths
18947+ bucket = stbir__horizontal_cost_bucket( horizontal_filter_pixel_width );
18948+ cost_ratio = (stbir_int64) ( ( horizontal_is_period1 ? stbir__horizontal_cost_period1 : stbir__horizontal_cost )[ channel_index ][ bucket ] * 1000.0f + 0.5f );
18949+
18950+ // the inputs with their wrap margins
18951+ horizontal_input = (stbir_int64) horizontal_input_size + ( ( horizontal_edge == STBIR_EDGE_WRAP ) ? horizontal_filter_pixel_width : 0 );
18952+ vertical_input = (stbir_int64) vertical_input_size + ( ( vertical_edge == STBIR_EDGE_WRAP ) ? vertical_filter_pixel_width : 0 );
18953+ horizontal_output = horizontal_output_size;
18954+ vertical_output = vertical_output_size;
18955+ vertical_taps = (stbir_int64) vertical_filter_pixel_width + 1;
18956+
18957+ vertical_first = stbir__products_le( vertical_taps * vertical_output, ( horizontal_input - horizontal_output ) * 1000,
18958+ cost_ratio * ( horizontal_input + horizontal_output ), vertical_input - vertical_output );
18959+
18960+ // the test hook: report the choice, and let the tests force it
18961+ stbir__chose_vertical_first = vertical_first;
18962+ if ( stbir__force_vertical_first )
18963+ vertical_first = ( stbir__force_vertical_first == 2 ) ? 1 : 0;
18964+
18965+ return vertical_first;
18966+}
18967+
18968+// layout lookups - must match stbir_internal_pixel_layout
18969+static unsigned char stbir__pixel_channels[] = {
18970+ 1,2,3,3,4, // 1ch, 2ch, rgb, bgr, 4ch
18971+ 4,4,4,4,2,2, // RGBA,BGRA,ARGB,ABGR,RA,AR
18972+ 4,4,4,4,2,2, // RGBA_PM,BGRA_PM,ARGB_PM,ABGR_PM,RA_PM,AR_PM
18973+};
18974+
18975+// the internal pixel layout enums are in a different order, so we can easily do range comparisons of types
18976+// the public pixel layout is ordered in a way that if you cast num_channels (1-4) to the enum, you get something sensible
18977+static stbir_internal_pixel_layout stbir__pixel_layout_convert_public_to_internal[] = {
18978+ STBIRI_BGR, STBIRI_1CHANNEL, STBIRI_2CHANNEL, STBIRI_RGB, STBIRI_RGBA,
18979+ STBIRI_4CHANNEL, STBIRI_BGRA, STBIRI_ARGB, STBIRI_ABGR, STBIRI_RA, STBIRI_AR,
18980+ STBIRI_RGBA_PM, STBIRI_BGRA_PM, STBIRI_ARGB_PM, STBIRI_ABGR_PM, STBIRI_RA_PM, STBIRI_AR_PM,
18981+};
18982+
18983+// We need a good guess at the L2 size for the column sizing (CACHE SIZE in the docs). We get this size, in order of preference as:
18984+// - stbir_l2_cache_size_override, if you set it to something non-zero before resizing (static, starts at 0, bytes)
18985+// - the OS, if you define one of these before including this header (we don't include the headers for you):
18986+// STBIR_ASK_WINDOWS_FOR_L2_SIZE - needs <windows.h> and <malloc.h> (GetLogicalProcessorInformationEx into an _alloca buffer)
18987+// STBIR_ASK_LINUX_FOR_L2_SIZE - needs <unistd.h> (sysconf _SC_LEVEL2_CACHE_SIZE)
18988+// STBIR_ASK_MACOS_FOR_L2_SIZE - needs <sys/sysctl.h> (sysctlbyname hw.l2cachesize)
18989+// - cpuid on x86/x64 (leaf 0x80000006, the L2 size in KB - Intel and AMD both fill it in)
18990+// - 256 KB if none of that applies or answers
18991+// An L2 of 1 MB or more is clamped to 512 KB to keep total resize internal memory use down.
18992+
18993+static int stbir_l2_cache_size_override = 0; // bytes, 0 = detect
18994+
18995+#if ( defined( _M_X64 ) || defined( _M_IX86 ) || defined( __x86_64__ ) || defined( __i386__ ) ) && !defined( _M_ARM64EC )
18996+ #define STBIR__L2_FROM_CPUID
18997+ #ifdef _MSC_VER
18998+ #include <intrin.h>
18999+ #else
19000+ #include <cpuid.h>
19001+ #endif
19002+#endif
19003+
19004+// (asked on every build - cpuid is a few hundred cycles, and no cache means nothing shared between threads building at once)
19005+static int stbir__l2_cache_size( void )
19006+{
19007+ int bytes = 0;
19008+
19009+ if ( stbir_l2_cache_size_override > 0 )
19010+ return stbir_l2_cache_size_override;
19011+
19012+ #if defined( STBIR_ASK_WINDOWS_FOR_L2_SIZE )
19013+ {
19014+ // the level 2 data or unified cache of the first processor group that reports one. the list is a few hundred
19015+ // bytes per cache level per core, so it goes on the stack (a 64 core machine's is ~30 KB; past 256 KB fall through)
19016+ DWORD len = 0;
19017+ GetLogicalProcessorInformationEx( RelationCache, 0, &len );
19018+ if ( ( len ) && ( len <= 256 * 1024 ) )
19019+ {
19020+ char * buf = (char*) _alloca( len );
19021+ if ( GetLogicalProcessorInformationEx( RelationCache, (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX*) buf, &len ) )
19022+ {
19023+ DWORD ofs = 0;
19024+ while ( ofs < len )
19025+ {
19026+ SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX * info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX*) ( buf + ofs );
19027+ if ( ( info->Relationship == RelationCache ) && ( info->Cache.Level == 2 ) && ( ( info->Cache.Type == CacheData ) || ( info->Cache.Type == CacheUnified ) ) )
19028+ {
19029+ bytes = (int) info->Cache.CacheSize;
19030+ break;
19031+ }
19032+ ofs += info->Size;
19033+ }
19034+ }
19035+ }
19036+ }
19037+ #elif defined( STBIR_ASK_LINUX_FOR_L2_SIZE )
19038+ {
19039+ long v = sysconf( _SC_LEVEL2_CACHE_SIZE );
19040+ if ( v > 0 )
19041+ bytes = (int) v;
19042+ }
19043+ #elif defined( STBIR_ASK_MACOS_FOR_L2_SIZE )
19044+ {
19045+ stbir_uint64 v = 0;
19046+ size_t vlen = sizeof( v );
19047+ if ( ( sysctlbyname( "hw.l2cachesize", &v, &vlen, 0, 0 ) == 0 ) && ( v > 0 ) )
19048+ bytes = (int) v;
19049+ }
19050+ #endif
19051+
19052+ #if defined( STBIR__L2_FROM_CPUID )
19053+ if ( bytes <= 0 ) // no OS query, or it didn't answer
19054+ {
19055+ unsigned int regs[4] = { 0, 0, 0, 0 };
19056+ #ifdef _MSC_VER
19057+ __cpuid( (int*) regs, 0x80000000 );
19058+ if ( regs[0] >= 0x80000006 )
19059+ {
19060+ __cpuid( (int*) regs, 0x80000006 );
19061+ bytes = (int) ( regs[2] >> 16 ) * 1024; // ECX[31:16] = L2 size in KB
19062+ }
19063+ #else
19064+ if ( __get_cpuid_max( 0x80000000, 0 ) >= 0x80000006 )
19065+ {
19066+ __get_cpuid( 0x80000006, ®s[0], ®s[1], ®s[2], ®s[3] );
19067+ bytes = (int) ( regs[2] >> 16 ) * 1024;
19068+ }
19069+ #endif
19070+ }
19071+ #endif
19072+
19073+ if ( bytes <= 0 )
19074+ bytes = 256 * 1024;
19075+ else if ( bytes >= 1024 * 1024 )
19076+ bytes = 512 * 1024;
19077+
19078+ return bytes;
19079+}
19080+
19081+// The splits (what a caller threads over) are a grid of split_cols column groups x split_rows row bands. Any grid gives
19082+// the same pixels and much the same memory (a split holds one slice's buffers either way), so stbir__choose_split_grid
19083+// picks the fastest by estimate, for every count from the request down to one split:
19084+// - a row band re-decodes the vertical filter's taps rows before its first output row, so short bands cost more.
19085+// - column groups are free while they are the slices a single thread would run anyway (the L2 fit); each extra slice
19086+// costs stbir_split_slice_row_cost columns of work per row. A group streams at least stbir_split_min_group_bytes
19087+// of each input row; below stbir_split_group_full_bytes it pays stbir_split_group_cost_pct per (full / bytes - 1)
19088+// per 8 threads, and narrow rows cost (threads / 8)^2 x (64 KB / input row bytes) / stbir_split_stream_cost_div.
19089+// - a thread costs stbir_thread_start_cost of work to start and join, and the count is capped so that the work
19090+// (output pixels x taps x channels / 4) is at least stbir_split_min_work x splits^2.
19091+// - each count is scored by its largest split, and the last threads are dropped while they add less than
19092+// stbir_split_min_thread_gain_pct of a thread's speed-up each over the best smaller count.
19093+// So the build may return fewer splits than asked for (which was always allowed); stbir_split_extra_threads adds some back.
19094+
19095+// the split estimate's tuning (statics so an app can override them before resizing, like stbir_l2_cache_size_override;
19096+// work units are output pixels x taps x channels / 4, roughly nanoseconds of resizing on a fast core)
19097+static int stbir_split_min_group_bytes = 1024; // the least input row a column group streams (bytes)
19098+static int stbir_split_group_full_bytes = 4096; // below this much input row a column group pays the narrow group cost:
19099+static int stbir_split_group_cost_pct = 30; // this percent x ( full / bytes - 1 ) x ( threads / 8 ) of the work
19100+static int stbir_split_stream_cost_div = 170; // narrow rows under many threads: ( threads / 8 )^2 x ( 64 KB / input row bytes ) / this, whatever the grid
19101+static int stbir_split_slice_row_cost = 250; // input columns' worth of work per row for each slice beyond the L2 fit
19102+static int stbir_thread_start_cost = 12000; // starting and joining one thread, in work units (a spinning pool ~1000, a thread per job 50000+)
19103+static int stbir_split_min_thread_gain_pct = 30; // the last threads must each add this percent of a thread's worth of speed-up, or they are dropped
19104+static int stbir_split_min_work = 20000; // the work must be at least this x splits^2
19105+static int stbir_split_extra_threads = 0; // use this many more splits than the estimate settles on (never more than asked for)
19106+#define STBIR__SPLIT_MAX_SPLITS 256 // the grid search's table
19107+
19108+
19109+// slice boundaries are multiples of this many pixels (so the SIMD loops only run their remnants at the end of the row)
19110+#define STBIR__SLICE_STEP 16
19111+
19112+// the most slices a row can have (each at least a step wide)
19113+static int stbir__slice_max_count( int output_columns )
19114+{
19115+ int n = output_columns / STBIR__SLICE_STEP;
19116+ return ( n < 1 ) ? 1 : n;
19117+}
19118+
19119+
19120+// The split grid for a request of *splits: cols x rows, and *splits becomes cols * rows (never more than asked for).
19121+static void stbir__choose_split_grid( int * splits, int * out_cols, int * out_rows, int mode, int output_w, int output_h, int input_w, int input_pixel_bytes, int taps_v, float inv_scale_v, int l2_slices, int effective_channels )
19122+{
19123+ int max_splits = *splits;
19124+ int requested_splits;
19125+ int max_cols = stbir__slice_max_count( output_w ); // a column group needs at least one slice
19126+ int best_cols = 1;
19127+ int best_rows = 1;
19128+
19129+ if ( max_splits < 1 )
19130+ max_splits = 1;
19131+ if ( max_splits > STBIR__SPLIT_MAX_SPLITS )
19132+ max_splits = STBIR__SPLIT_MAX_SPLITS;
19133+ requested_splits = max_splits;
19134+
19135+ if ( stbir__split_force_cols > 0 )
19136+ {
19137+ // Debug: a forced grid.
19138+ best_cols = ( stbir__split_force_cols < max_cols ) ? stbir__split_force_cols : max_cols;
19139+ if ( best_cols > max_splits )
19140+ best_cols = max_splits;
19141+ best_rows = max_splits / best_cols;
19142+ }
19143+ else if ( mode == 1 )
19144+ {
19145+ // Row bands only.
19146+ best_rows = ( max_splits < output_h ) ? max_splits : output_h;
19147+ }
19148+ else if ( mode == 2 )
19149+ {
19150+ // Column groups only.
19151+ best_cols = ( max_splits < max_cols ) ? max_splits : max_cols;
19152+ }
19153+ else
19154+ {
19155+ // Everything in fixed point: times are in 1/65536ths of the single threaded work.
19156+ stbir_uint64 work = (stbir_uint64)output_w * (stbir_uint64)output_h * (stbir_uint64)taps_v * (stbir_uint64)effective_channels / 4;
19157+ stbir_uint64 band_warmup = (stbir_uint64)( (float)taps_v * 65536.0f / ( ( inv_scale_v > 1.0f ) ? inv_scale_v : 1.0f ) ); // what a band re-decodes before its first row, in output rows' worth of work
19158+ stbir_uint64 row_bytes = (stbir_uint64)input_w * (stbir_uint64)input_pixel_bytes;
19159+ stbir_uint64 estimate_of[ STBIR__SPLIT_MAX_SPLITS + 1 ];
19160+ int cols_of[ STBIR__SPLIT_MAX_SPLITS + 1 ];
19161+ int rows_of[ STBIR__SPLIT_MAX_SPLITS + 1 ];
19162+ int count, cols;
19163+
19164+ if ( work < 1 )
19165+ work = 1;
19166+ if ( row_bytes < 1 )
19167+ row_bytes = 1;
19168+
19169+ // The thread start/join cost grows with the count: the work must be at least min_work x splits^2.
19170+ while ( ( max_splits > 1 ) && ( work < (stbir_uint64)stbir_split_min_work * (stbir_uint64)max_splits * (stbir_uint64)max_splits ) )
19171+ --max_splits;
19172+
19173+ // The best grid for every count up to the request, by estimated time (the largest split's - it sets the pace).
19174+ for (count = 1; count <= requested_splits; count++)
19175+ {
19176+ estimate_of[count] = ~(stbir_uint64)0;
19177+ cols_of[count] = 1;
19178+ rows_of[count] = 1;
19179+
19180+ for (cols = 1; cols <= count; cols++)
19181+ {
19182+ int rows, rows_per_band, extra_slices;
19183+ stbir_uint64 group_bytes, narrow_cost, stream_cost, per_row_cost, estimate;
19184+
19185+ if ( count % cols )
19186+ continue;
19187+ if ( cols > max_cols )
19188+ break;
19189+
19190+ // A column group streams its share of each input row, and never less than min_group_bytes of it.
19191+ group_bytes = row_bytes / (stbir_uint64)cols;
19192+ if ( group_bytes < 1 )
19193+ group_bytes = 1;
19194+ if ( ( cols > 1 ) && ( group_bytes < (stbir_uint64)stbir_split_min_group_bytes ) )
19195+ break;
19196+
19197+ // Narrow groups cost group_cost_pct per ( full / bytes - 1 ) per 8 threads.
19198+ narrow_cost = 0;
19199+ if ( ( cols > 1 ) && ( group_bytes < (stbir_uint64)stbir_split_group_full_bytes ) )
19200+ narrow_cost = ( ( (stbir_uint64)stbir_split_group_full_bytes * 65536 ) / group_bytes - 65536 ) * (stbir_uint64)stbir_split_group_cost_pct * (stbir_uint64)count / ( 100 * 8 );
19201+
19202+ // Narrow rows under many threads cost the same whatever the grid (every row is a fresh page, latency bound, and
19203+ // the latency climbs with the thread count): the larger of this and the narrow group cost.
19204+ stream_cost = ( (stbir_uint64)count * (stbir_uint64)count * 65536 * 65536 ) / ( (stbir_uint64)64 * (stbir_uint64)stbir_split_stream_cost_div * row_bytes );
19205+ if ( stream_cost > narrow_cost )
19206+ narrow_cost = stream_cost;
19207+
19208+ rows = count / cols;
19209+ if ( rows > output_h )
19210+ continue;
19211+
19212+ // The slowest split sets the time: the largest band (rows rounded up) doing the largest group's share of the
19213+ // slices (the slice count rounded up to a multiple of the groups), against an even share of the work.
19214+ rows_per_band = ( output_h + rows - 1 ) / rows;
19215+ extra_slices = ( ( l2_slices + cols - 1 ) / cols ) * cols - l2_slices;
19216+ per_row_cost = 65536 + band_warmup / (stbir_uint64)rows_per_band + ( (stbir_uint64)extra_slices * (stbir_uint64)stbir_split_slice_row_cost * 65536 ) / (stbir_uint64)input_w + narrow_cost;
19217+ estimate = ( (stbir_uint64)rows_per_band * (stbir_uint64)rows ) * per_row_cost / ( (stbir_uint64)output_h * (stbir_uint64)count );
19218+ estimate += ( (stbir_uint64)count * (stbir_uint64)stbir_thread_start_cost * 65536 ) / work; // plus the threads' own start/join cost
19219+
19220+ if ( estimate < estimate_of[count] )
19221+ {
19222+ estimate_of[count] = estimate;
19223+ cols_of[count] = cols;
19224+ rows_of[count] = rows;
19225+ }
19226+ }
19227+ }
19228+
19229+ // From the request down, drop threads while they add less than min_thread_gain_pct of a thread's speed-up each
19230+ // over the best smaller count (not the next one down: a prime count only has bad grids, which makes the count
19231+ // above it look good).
19232+ count = max_splits;
19233+ while ( count > 1 )
19234+ {
19235+ stbir_uint64 speedup = ( estimate_of[1] * 256 ) / estimate_of[count];
19236+ stbir_uint64 best_speedup = 0;
19237+ int fewer, best_fewer = 1;
19238+
19239+ for (fewer = 1; fewer < count; fewer++)
19240+ {
19241+ stbir_uint64 speedup_fewer = ( estimate_of[1] * 256 ) / estimate_of[fewer];
19242+ if ( speedup_fewer > best_speedup )
19243+ {
19244+ best_speedup = speedup_fewer;
19245+ best_fewer = fewer;
19246+ }
19247+ }
19248+
19249+ if ( ( speedup > best_speedup ) && ( ( speedup - best_speedup ) * 100 >= (stbir_uint64)stbir_split_min_thread_gain_pct * 256 * (stbir_uint64)( count - best_fewer ) ) )
19250+ break;
19251+
19252+ // The extra threads over the best smaller count weren't worth it: fall back to it and check again.
19253+ count = best_fewer;
19254+ }
19255+
19256+ // The app wants more than the estimate settled on.
19257+ if ( stbir_split_extra_threads > 0 )
19258+ {
19259+ count += stbir_split_extra_threads;
19260+ if ( count > requested_splits )
19261+ count = requested_splits;
19262+ // A count no grid fits (more bands than rows): the next one down.
19263+ while ( ( count > 1 ) && ( estimate_of[count] == ~(stbir_uint64)0 ) )
19264+ --count;
19265+ }
19266+
19267+ best_cols = cols_of[count];
19268+ best_rows = rows_of[count];
19269+ }
19270+
19271+ if ( best_cols > max_cols )
19272+ best_cols = max_cols;
19273+ if ( best_rows > output_h )
19274+ best_rows = output_h;
19275+ if ( best_cols < 1 )
19276+ best_cols = 1;
19277+ if ( best_rows < 1 )
19278+ best_rows = 1;
19279+
19280+ *out_cols = best_cols;
19281+ *out_rows = best_rows;
19282+ *splits = best_cols * best_rows;
19283+}
19284+
19285+// where slice number slice of num_slices ends
19286+static int stbir__slice_end( int output_columns, int num_slices, int slice )
19287+{
19288+ stbir_uint64 end;
19289+
19290+ if ( slice >= num_slices - 1 )
19291+ return output_columns;
19292+
19293+ end = ( (stbir_uint64)( slice + 1 ) * (stbir_uint64)output_columns ) / (stbir_uint64)num_slices;
19294+ return ( (int)end / STBIR__SLICE_STEP ) * STBIR__SLICE_STEP;
19295+}
19296+
19297+// the widest slice there can be for a count (for sizing the per slice buffers): the average plus a step of rounding
19298+static int stbir__slice_columns_for( int output_columns, int slices )
19299+{
19300+ int columns = ( output_columns + slices - 1 ) / slices + STBIR__SLICE_STEP;
19301+ if ( columns >= output_columns )
19302+ columns = output_columns;
19303+ return columns;
19304+}
19305+
19306+// the input pixels a slice decodes for that many columns (see the sizing in stbir__alloc_internal_mem_and_build_samplers)
19307+static int stbir__slice_in_span_for( stbir__sampler const * horizontal, int slice_columns, int conservative_in_span )
19308+{
19309+ int span = (int)( slice_columns * horizontal->scale_info.inv_scale ) + 1 + horizontal->filter_pixel_width + 2 * horizontal->coefficient_width + 4;
19310+ if ( ( slice_columns >= horizontal->scale_info.output_sub_size ) || ( span > conservative_in_span ) )
19311+ span = conservative_in_span;
19312+ return span;
19313+}
19314+
19315+// the ring buffer for one slice: its entries are horizontal output rows (slice_columns wide), or decoded input rows
19316+// (slice_in_span wide) when vertical first
19317+static size_t stbir__ring_bytes_for( stbir__sampler const * horizontal, int slice_columns, int conservative_in_span, int effective_channels, int vertical_first, int ring_entries )
19318+{
19319+ int row_pixels = vertical_first ? stbir__slice_in_span_for( horizontal, slice_columns, conservative_in_span ) : slice_columns;
19320+ return (size_t)ring_entries * ( (size_t)row_pixels * effective_channels * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING );
19321+}
19322+
19323+// pick the slice count: the fewest slices whose ring buffer fits the L2 budget (allowing an eighth over - a little
19324+// spill costs less than another slice's re-decode), or the most slices possible (16 columns each) if none fit
19325+static int stbir__choose_column_slices( stbir__sampler const * horizontal, int conservative_in_span, int effective_channels, int vertical_first, int ring_entries )
19326+{
19327+ int output_columns = horizontal->scale_info.output_sub_size;
19328+ size_t budget = (size_t)stbir__l2_cache_size();
19329+ size_t allowed = budget + budget / 8;
19330+ int max_slices = stbir__slice_max_count( output_columns );
19331+ int slices;
19332+
19333+ for (slices = 1; slices < max_slices; slices++)
19334+ {
19335+ int slice_columns = stbir__slice_columns_for( output_columns, slices );
19336+ if ( stbir__ring_bytes_for( horizontal, slice_columns, conservative_in_span, effective_channels, vertical_first, ring_entries ) <= allowed )
19337+ return slices;
19338+ }
19339+ return max_slices;
19340+}
19341+
19342+// splits the output columns into num_column_slices balanced slices on step boundaries (stbir__slice_end), and works
19343+// out the input range each slice has to decode: the lowest n0 through the furthest pixel any of its kernels read
19344+// (stbir__pack_coefficients arranged for those reads to stay within the conservative range).
19345+static void stbir__build_column_slices( stbir__info * info, int slice_columns, int in_span_bound )
19346+{
19347+ stbir__sampler const * horizontal = &info->horizontal;
19348+ int output_columns = horizontal->scale_info.output_sub_size;
19349+ int widest = horizontal->coefficient_width; // the coefficients are packed to widest by now
19350+ int first_column, end_column;
19351+ int slice_index = 0;
19352+
19353+ STBIR__UNUSED( in_span_bound ); // both only checked in asserts
19354+ STBIR__UNUSED( slice_columns );
19355+
19356+ for (first_column = 0; first_column < output_columns; first_column = end_column, slice_index++)
19357+ {
19358+ stbir__column_slice * slice = info->column_slices + slice_index;
19359+ int in_first = 0x7fffffff;
19360+ int in_last = -0x7fffffff;
19361+ int x, unused;
19362+
19363+ end_column = stbir__slice_end( output_columns, info->num_column_slices, slice_index );
19364+ STBIR_ASSERT( ( end_column > first_column ) && ( end_column <= output_columns ) && ( ( end_column - first_column ) <= slice_columns ) );
19365+
19366+ // the lowest first pixel through the furthest pixel any of the slice's kernels read
19367+ for (x = first_column; x < end_column; x++)
19368+ {
19369+ stbir__contributors contributor;
19370+ int last_read;
19371+
19372+ stbir__output_range( horizontal, x, &contributor.n0, &contributor.n1 );
19373+ last_read = contributor.n0 + stbir__horizontal_read_extent( &contributor, widest ) - 1;
19374+ if ( contributor.n0 < in_first )
19375+ in_first = contributor.n0;
19376+ if ( last_read > in_last )
19377+ in_last = last_read;
19378+ }
19379+
19380+ STBIR_ASSERT( in_first >= info->conservative_extents.n0 );
19381+ STBIR_ASSERT( in_last <= info->conservative_extents.n1 );
19382+ STBIR_ASSERT( ( in_last - in_first + 1 ) <= in_span_bound ); // the buffers were sized from in_span_bound in the first allocation pass
19383+
19384+ slice->x0 = first_column;
19385+ slice->width = end_column - first_column;
19386+ slice->in_n0 = in_first;
19387+ slice->in_n1 = in_last;
19388+ stbir__output_range( horizontal, first_column, &slice->first_n0, &unused );
19389+ }
19390+
19391+ STBIR_ASSERT( slice_index == info->num_column_slices );
19392+}
19393+
19394+// split s is row band ( s / split_cols ) of column group ( s % split_cols ).
19395+static void stbir__setup_split_grid( stbir__info * info, int split_cols, int split_rows )
19396+{
19397+ int r, c, next = 0;
19398+
19399+ info->split_cols = split_cols;
19400+ info->split_rows = split_rows;
19401+ stbir__get_split_info( info->split_info, split_rows, info->vertical.scale_info.output_sub_size, info->vertical.filter_pixel_margin, info->vertical.scale_info.input_full_size, info->vertical.is_gather, &info->vertical );
19402+ for( r = split_rows - 1 ; r >= 0 ; r-- )
19403+ {
19404+ stbir__per_split_info band = info->split_info[r];
19405+ for( c = split_cols - 1 ; c >= 0 ; c-- )
19406+ {
19407+ stbir__per_split_info * s = info->split_info + r * split_cols + c;
19408+ s->start_output_y = band.start_output_y;
19409+ s->end_output_y = band.end_output_y;
19410+ s->start_input_y = band.start_input_y;
19411+ s->end_input_y = band.end_input_y;
19412+ }
19413+ }
19414+ for( c = 0 ; c < split_cols ; c++ )
19415+ {
19416+ int count = ( info->num_column_slices - next ) / ( split_cols - c );
19417+ for( r = 0 ; r < split_rows ; r++ )
19418+ {
19419+ info->split_info[ r * split_cols + c ].first_column_slice = next;
19420+ info->split_info[ r * split_cols + c ].num_column_slices = count;
19421+ }
19422+ next += count;
19423+ }
19424+}
19425+
19426+// the per split buffers get a page of slack, to be repositioned within it at resize time (stbir__4K_alias_skew_buffers)
19427+#define STBIR__4K_ALIAS_SKEW_SLACK 4096
19428+
19429+// What kind of alpha weighting the layouts call for: 0 = none, 1 = weight in, leave premultiplied, 2 = fancy weight
19430+// in and unweight out, 3 = premultiplied in, unweight out, 4 = simple weight in and unweight out. Fancy weighting
19431+// carries extra channels through the filters (*effective_channels).
19432+static int stbir__alpha_weighting_type( stbir__sampler const * horizontal, stbir__sampler const * vertical, stbir_internal_pixel_layout input_pixel_layout, stbir_internal_pixel_layout output_pixel_layout, int fast_alpha, int * effective_channels )
19433+{
19434+ int input_has_alpha = ( input_pixel_layout >= STBIRI_RGBA ) && ( input_pixel_layout <= STBIRI_AR );
19435+ int input_premultiplied = ( input_pixel_layout >= STBIRI_RGBA_PM ) && ( input_pixel_layout <= STBIRI_AR_PM );
19436+ int output_has_alpha = ( output_pixel_layout >= STBIRI_RGBA ) && ( output_pixel_layout <= STBIRI_AR );
19437+ int output_premultiplied = ( output_pixel_layout >= STBIRI_RGBA_PM ) && ( output_pixel_layout <= STBIRI_AR_PM );
19438+
19439+ int point_sampled = ( horizontal->filter_enum == STBIR_FILTER_POINT_SAMPLE ) && ( vertical->filter_enum == STBIR_FILTER_POINT_SAMPLE );
19440+
19441+ if ( input_has_alpha && output_has_alpha )
19442+ {
19443+ static int fancy_alpha_effective_cnts[6] = { 7, 7, 7, 7, 3, 3 };
19444+ if ( point_sampled ) // nothing is blended, so weighting in and unweighting out would be the identity
19445+ return 0;
19446+ if ( fast_alpha )
19447+ return 4;
19448+ *effective_channels = fancy_alpha_effective_cnts[ input_pixel_layout - STBIRI_RGBA ];
19449+ return 2;
19450+ }
19451+ // (the conversions between premultiplied and not still happen when point sampling)
19452+ if ( input_premultiplied && output_has_alpha )
19453+ return 3;
19454+ if ( input_has_alpha && output_premultiplied )
19455+ return 1;
19456+ return 0;
19457+}
19458+
19459+// The alpha weight and unweight functions for the weighting type (and the 3 channel color flip, which rides the same path).
19460+static void stbir__set_alpha_functions( stbir__info * info, int alpha_weighting_type, stbir_internal_pixel_layout input_pixel_layout, stbir_internal_pixel_layout output_pixel_layout, float horizontal_scale )
19461+{
19462+ static stbir__alpha_weight_func * fancy_alpha_weights[6] = { stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_2ch, stbir__fancy_alpha_weight_2ch };
19463+ static stbir__alpha_unweight_func * fancy_alpha_unweights[6] = { stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_2ch, stbir__fancy_alpha_unweight_2ch };
19464+ static stbir__alpha_weight_func * simple_alpha_weights[6] = { stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_2ch, stbir__simple_alpha_weight_2ch };
19465+ static stbir__alpha_unweight_func * simple_alpha_unweights[6] = { stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_2ch, stbir__simple_alpha_unweight_2ch };
19466+
19467+ info->alpha_weight = 0;
19468+ info->alpha_unweight = 0;
19469+
19470+ if ( alpha_weighting_type == 2 )
19471+ {
19472+ // high quality alpha multiplying on the way in, dividing on the way out
19473+ info->alpha_weight = fancy_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
19474+ info->alpha_unweight = fancy_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
19475+ }
19476+ else if ( alpha_weighting_type == 4 )
19477+ {
19478+ // fast alpha multiplying on the way in, dividing on the way out
19479+ info->alpha_weight = simple_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
19480+ info->alpha_unweight = simple_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
19481+ }
19482+ else if ( alpha_weighting_type == 1 )
19483+ {
19484+ // fast alpha on the way in, leave in premultiplied form on way out
19485+ info->alpha_weight = simple_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
19486+ }
19487+ else if ( alpha_weighting_type == 3 )
19488+ {
19489+ // incoming is premultiplied, fast alpha dividing on the way out - non-premultiplied output
19490+ info->alpha_unweight = simple_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
19491+ }
19492+
19493+ // handle 3-chan color flipping, using the alpha weight path
19494+ if ( ( ( input_pixel_layout == STBIRI_RGB ) && ( output_pixel_layout == STBIRI_BGR ) ) ||
19495+ ( ( input_pixel_layout == STBIRI_BGR ) && ( output_pixel_layout == STBIRI_RGB ) ) )
19496+ {
19497+ // do the flipping on the smaller of the two ends
19498+ if ( horizontal_scale < 1.0f )
19499+ info->alpha_unweight = stbir__simple_flip_3ch;
19500+ else
19501+ info->alpha_weight = stbir__simple_flip_3ch;
19502+ }
19503+}
19504+
19505+// Shared coefficients require identical geometry; inactive rational and exact-position fields may be uninitialized.
19506+static int stbir__scale_info_matches( stbir__scale_info const * horizontal, stbir__scale_info const * vertical )
19507+{
19508+ if ( ( horizontal->input_full_size != vertical->input_full_size ) || ( horizontal->output_sub_size != vertical->output_sub_size ) ||
19509+ ( horizontal->scale != vertical->scale ) || ( horizontal->inv_scale != vertical->inv_scale ) || // (the positions come from the reals)
19510+ ( horizontal->pixel_shift != vertical->pixel_shift ) || ( horizontal->scale_is_rational != vertical->scale_is_rational ) ||
19511+ ( horizontal->positions_exact != vertical->positions_exact ) )
19512+ {
19513+ return 0;
19514+ }
19515+
19516+ if ( horizontal->scale_is_rational &&
19517+ ( ( horizontal->scale_numerator != vertical->scale_numerator ) || ( horizontal->scale_denominator != vertical->scale_denominator ) ) )
19518+ {
19519+ return 0;
19520+ }
19521+
19522+ if ( horizontal->positions_exact &&
19523+ ( ( horizontal->shift_pixels != vertical->shift_pixels ) || ( horizontal->tap_scale_in != vertical->tap_scale_in ) ||
19524+ ( horizontal->tap_scale_out != vertical->tap_scale_out ) ) )
19525+ {
19526+ return 0;
19527+ }
19528+
19529+ return 1;
19530+}
19531+
19532+// Are the two samplers identical? (happens a lot with mipmap generation) Then the vertical can share the horizontal's tables.
19533+static int stbir__samplers_match( stbir__sampler const * horizontal, stbir__sampler const * vertical )
19534+{
19535+ return ( horizontal->filter_kernel_new == vertical->filter_kernel_new ) && ( horizontal->filter_kernel_old == vertical->filter_kernel_old ) && ( horizontal->filter_support_new == vertical->filter_support_new ) && ( horizontal->filter_radius == vertical->filter_radius ) && ( horizontal->filter_support_old == vertical->filter_support_old ) &&
19536+ ( horizontal->edge == vertical->edge ) && ( horizontal->is_gather == vertical->is_gather ) &&
19537+ stbir__scale_info_matches( &horizontal->scale_info, &vertical->scale_info );
19538+}
19539+
19540+// The horizontal kernels for this channel count and coefficient width.
19541+static void stbir__select_horizontal_kernels( stbir__info * info, stbir__sampler const * horizontal, int effective_channels )
19542+{
19543+ int widest = horizontal->extent_info.widest;
19544+
19545+ // start with defaulting to the n_coeffs functions (specialized on channels and remnant leftover)
19546+ info->horizontal_gather_channels = stbir__horizontal_gather_n_coeffs_funcs[ effective_channels ][ widest & 3 ];
19547+
19548+ // but if the number of coeffs <= 17, use another set of special cases. This is by far the common case
19549+ if ( widest <= 17 )
19550+ info->horizontal_gather_channels = stbir__horizontal_gather_channels_funcs[ effective_channels ][ widest - 1 ];
19551+
19552+ // integer downsamples (polyphase period 1) use one coefficient set for the whole interior, so the kernel can keep the
19553+ // coefficients in registers for the whole run - where there is one for this channel count and width (null otherwise)
19554+ info->horizontal_gather_period1 = 0;
19555+ if ( ( horizontal->shared_period == 1 ) && ( widest <= 17 ) )
19556+ info->horizontal_gather_period1 = stbir__horizontal_gather_period1_funcs[ effective_channels ][ widest ];
19557+}
19558+
19559+// The per split buffers are sized for one column slice (the slices run one after another and reuse them).
19560+typedef struct stbir__slice_buffer_sizes
19561+{
19562+ int num_column_slices; // a multiple of the column groups, so they get equal shares
19563+ int slice_columns; // the widest slice
19564+ int slice_in_span; // the most input pixels a slice decodes
19565+ size_t decode_buffer_size;
19566+ size_t ring_buffer_length_bytes; // one ring entry
19567+ size_t ring_buffer_size;
19568+ size_t vertical_buffer_size;
19569+} stbir__slice_buffer_sizes;
19570+
19571+static void stbir__size_slice_buffers( stbir__slice_buffer_sizes * sizes, stbir__sampler const * horizontal, int conservative_in_span, int l2_slices, int split_cols, int effective_channels, int vertical_first, int ring_buffer_num_entries )
19572+{
19573+ int output_columns = horizontal->scale_info.output_sub_size;
19574+ int max_slices = stbir__slice_max_count( output_columns );
19575+
19576+ // the slice count made a multiple of the column groups (a slice is at least a step wide, so some groups may get none)
19577+ sizes->num_column_slices = ( ( l2_slices + split_cols - 1 ) / split_cols ) * split_cols;
19578+ if ( sizes->num_column_slices > max_slices )
19579+ sizes->num_column_slices = max_slices;
19580+ sizes->slice_columns = stbir__slice_columns_for( output_columns, sizes->num_column_slices );
19581+
19582+ // the input pixels a slice decodes: its columns' geometric span, plus the filter width, plus the room
19583+ // stbir__pack_coefficients can move a contributor back by and the kernel can read past it (a coefficient
19584+ // width each). the exact ranges come from the packed contributors in the second pass, which asserts
19585+ // they fit in this. a single slice is exactly the conservative range.
19586+ sizes->slice_in_span = stbir__slice_in_span_for( horizontal, sizes->slice_columns, conservative_in_span );
19587+
19588+ // sometimes read one float off in some of the unrolled loops (with a weight of zero coeff, so it doesn't have an effect)
19589+ // we use a few extra floats instead of just 1, so that input callback buffer can overlap with the decode buffer without
19590+ // the conversion routines overwriting the callback input data.
19591+ sizes->decode_buffer_size = (size_t)sizes->slice_in_span * effective_channels * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // extra floats for input callback stagger
19592+
19593+#if defined( STBIR__SEPARATE_ALLOCATIONS ) && defined(STBIR_SIMD8)
19594+ if ( effective_channels == 3 )
19595+ sizes->decode_buffer_size += sizeof(float); // avx in 3 channel mode needs one float at the start of the buffer (only with separate allocations)
19596+#endif
19597+
19598+ sizes->ring_buffer_length_bytes = (size_t)sizes->slice_columns * (size_t)effective_channels * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // extra floats for padding
19599+
19600+ // if we do vertical first, the ring buffer holds a whole decoded line
19601+ if ( vertical_first )
19602+ sizes->ring_buffer_length_bytes = ( sizes->decode_buffer_size + 15 ) & ~15;
19603+
19604+ // keep the ring buffer's entries off a page multiple apart: the vertical gather reads taps rows at the same column,
19605+ // and at a 4K multiple stride they all fall in the same cache sets
19606+ if ( ( sizes->ring_buffer_length_bytes & 4095 ) == 0 )
19607+ sizes->ring_buffer_length_bytes += 64*3;
19608+
19609+ sizes->ring_buffer_size = (size_t)ring_buffer_num_entries * sizes->ring_buffer_length_bytes;
19610+
19611+ // The vertical buffer holds one horizontally resampled row: gathering, the output row (the vertical gather's result
19612+ // horizontal first, the horizontal's result vertical first); scattering, the input row about to be scattered
19613+ // (horizontal first) or the output row at eviction (vertical first). The accumulation itself is in the ring.
19614+ sizes->vertical_buffer_size = (size_t)sizes->slice_columns * (size_t)effective_channels * sizeof(float) + sizeof(float); // extra float for padding
19615+}
19616+
19617+static stbir__info * stbir__alloc_internal_mem_and_build_samplers( stbir__sampler * horizontal, stbir__sampler * vertical, stbir__contributors * conservative, stbir_pixel_layout input_pixel_layout_public, stbir_pixel_layout output_pixel_layout_public, int splits, int new_x, int new_y, int fast_alpha, int input_pixel_bytes, void * user_data STBIR_ONLY_PROFILE_BUILD_GET_INFO )
19618+{
19619+ static char stbir_channel_count_index[8]={ 9,0,1,2, 3,9,9,4 };
19620+
19621+ stbir__info * info = 0;
19622+ void * alloced = 0;
19623+ size_t alloced_total = 0;
19624+ int vertical_first;
19625+ int alloc_ring_buffer_num_entries;
19626+ int alpha_weighting_type;
19627+ int conservative_split_output_size;
19628+ int conservative_in_span, l2_slices, split_cols, split_rows;
19629+ int copy_horizontal;
19630+ stbir__slice_buffer_sizes sizes;
19631+ stbir_internal_pixel_layout input_pixel_layout = stbir__pixel_layout_convert_public_to_internal[ input_pixel_layout_public ];
19632+ stbir_internal_pixel_layout output_pixel_layout = stbir__pixel_layout_convert_public_to_internal[ output_pixel_layout_public ];
19633+ int channels = stbir__pixel_channels[ input_pixel_layout ];
19634+ int effective_channels = channels;
19635+
19636+ // channel in and out count must match currently
19637+ if ( channels != stbir__pixel_channels[ output_pixel_layout ] )
19638+ return 0;
19639+
19640+ alpha_weighting_type = stbir__alpha_weighting_type( horizontal, vertical, input_pixel_layout, output_pixel_layout, fast_alpha, &effective_channels );
19641+
19642+ // do the horizontal and vertical store their polyphase coefficient sets once? (sizes the tables, so before anything
19643+ // is allocated). the vertical only when gathering (the scatter tables are per input scanline). the vertical kernels
19644+ // read exactly n0..n1 of a set, so its epilogue needs no clip zone.
19645+ stbir__plan_polyphase_sharing( horizontal, 1, user_data );
19646+ if ( vertical->is_gather )
19647+ stbir__plan_polyphase_sharing( vertical, 0, user_data );
19648+
19649+ vertical_first = stbir__should_do_vertical_first( (int)stbir_channel_count_index[ effective_channels ],
19650+ horizontal->filter_pixel_width, horizontal->scale_info.input_full_size, horizontal->scale_info.output_sub_size, horizontal->edge, horizontal->shared_period == 1,
19651+ vertical->filter_pixel_width, vertical->scale_info.input_full_size, vertical->scale_info.output_sub_size, vertical->edge );
19652+
19653+ // ring rows: gathering, an output's taps rows (plus one for float rounding); scattering, the output rows an input
19654+ // row touches - 2 x the filter's support in output pixels, plus one, plus one for rounding
19655+ if ( vertical->is_gather )
19656+ alloc_ring_buffer_num_entries = vertical->filter_pixel_width + 1;
19657+ else
19658+ alloc_ring_buffer_num_entries = stbir__ceil_to_int( (stbir_real)2.0 * stbir__filter_radius_in_output_pixels( vertical->filter_radius, vertical->filter_support_new, vertical->filter_support_old, &vertical->scale_info, user_data ) ) + 2;
19659+
19660+ // how many vertical slices the single threaded resize would run (the setting, or the L2 fit), then the split grid -
19661+ // which can lower the split count
19662+ conservative_in_span = conservative->n1 - conservative->n0 + 1;
19663+ l2_slices = stbir__column_slices;
19664+ if ( l2_slices < 1 )
19665+ l2_slices = stbir__choose_column_slices( horizontal, conservative_in_span, effective_channels, vertical_first, alloc_ring_buffer_num_entries );
19666+ stbir__choose_split_grid( &splits, &split_cols, &split_rows, stbir__split_mode, horizontal->scale_info.output_sub_size, vertical->scale_info.output_sub_size, conservative_in_span, input_pixel_bytes, vertical->filter_pixel_width, (float) vertical->scale_info.inv_scale, l2_slices, effective_channels );
19667+ conservative_split_output_size = stbir__get_max_split( split_rows, vertical->scale_info.output_sub_size );
19668+
19669+ // we never need more ring buffer entries than the scanlines we're outputting when in scatter mode
19670+ if ( ( !vertical->is_gather ) && ( alloc_ring_buffer_num_entries > conservative_split_output_size ) )
19671+ alloc_ring_buffer_num_entries = conservative_split_output_size;
19672+
19673+ stbir__size_slice_buffers( &sizes, horizontal, conservative_in_span, l2_slices, split_cols, effective_channels, vertical_first, alloc_ring_buffer_num_entries );
19674+
19675+ // are the two filters identical? then the vertical shares the horizontal's tables instead of generating its own
19676+ copy_horizontal = stbir__samplers_match( horizontal, vertical );
19677+
19678+ // we make two passes through this loop, 1st to add everything up, 2nd to allocate and init
19679+ for(;;)
19680+ {
19681+ int i;
19682+ void * advance_mem = alloced;
19683+
19684+#ifdef STBIR__SEPARATE_ALLOCATIONS
19685+ // a failed allocation frees whatever got allocated: the pointers the cleanup frees are cleared as their holders are
19686+ // allocated, so every one is an allocation or null (if info itself failed, only the placeholder block exists)
19687+ #define STBIR__NEXT_PTR( ptr, size, ntype ) if ( alloced ) { void * p = STBIR_MALLOC( size, user_data); if ( p == 0 ) { if ( info ) stbir__free_internal_mem( info ); else STBIR_FREE( alloced, user_data ); return 0; } (ptr) = (ntype*)p; }
19688+#else
19689+ #define STBIR__NEXT_PTR( ptr, size, ntype ) advance_mem = (void*) ( ( ((size_t)advance_mem) + 15 ) & ~15 ); if ( alloced ) ptr = (ntype*)advance_mem; advance_mem = (char*)(((size_t)advance_mem) + (size));
19690+#endif
19691+
19692+ STBIR__NEXT_PTR( info, sizeof( stbir__info ), stbir__info );
19693+ if ( info )
19694+ {
19695+ info->alloced_mem = alloced;
19696+ info->user_data = user_data;
19697+ info->splits = splits;
19698+ info->alloc_ring_buffer_num_entries = (int)alloc_ring_buffer_num_entries;
19699+ info->effective_channels = effective_channels;
19700+ info->split_info = 0;
19701+ info->column_slices = 0;
19702+ info->horizontal.contributors = 0;
19703+ info->horizontal.coefficients = 0;
19704+ info->vertical.contributors = 0;
19705+ info->vertical.coefficients = 0;
19706+ }
19707+
19708+ STBIR__NEXT_PTR( info->split_info, sizeof( stbir__per_split_info ) * splits, stbir__per_split_info );
19709+ #ifdef STBIR__SEPARATE_ALLOCATIONS
19710+ if ( info )
19711+ {
19712+ for( i = 0 ; i < splits ; i++ )
19713+ {
19714+ info->split_info[i].ring_buffers = 0;
19715+ info->split_info[i].decode_buffer_alloc = 0;
19716+ info->split_info[i].vertical_buffer_alloc = 0;
19717+ }
19718+ }
19719+ #endif
19720+
19721+ if ( info )
19722+ {
19723+ // initialize info fields
19724+ info->alloced_total = alloced_total;
19725+ info->channels = channels;
19726+ info->offset_x = new_x;
19727+ info->offset_y = new_y;
19728+ info->ring_buffer_num_entries = 0;
19729+ info->ring_buffer_length_bytes = sizes.ring_buffer_length_bytes;
19730+ info->vertical_first = vertical_first;
19731+
19732+ info->input_pixel_layout_internal = input_pixel_layout;
19733+ info->output_pixel_layout_internal = output_pixel_layout;
19734+
19735+ stbir__set_alpha_functions( info, alpha_weighting_type, input_pixel_layout, output_pixel_layout, (float) horizontal->scale_info.scale );
19736+ }
19737+
19738+ // get all the per-split buffers (each with a page of slack, see stbir__4K_alias_skew_buffers)
19739+ for( i = 0 ; i < splits ; i++ )
19740+ {
19741+ STBIR__NEXT_PTR( info->split_info[i].decode_buffer, sizes.decode_buffer_size + STBIR__4K_ALIAS_SKEW_SLACK, float );
19742+
19743+#ifdef STBIR__SEPARATE_ALLOCATIONS
19744+
19745+ #ifdef STBIR_SIMD8
19746+ if ( ( info ) && ( effective_channels == 3 ) )
19747+ ++info->split_info[i].decode_buffer; // avx in 3 channel mode needs one float at the start of the buffer
19748+ #endif
19749+ if ( info )
19750+ info->split_info[i].decode_buffer_alloc = info->split_info[i].decode_buffer;
19751+
19752+ STBIR__NEXT_PTR( info->split_info[i].ring_buffers, alloc_ring_buffer_num_entries * sizeof(float*), float* );
19753+ {
19754+ int j;
19755+ if ( info )
19756+ {
19757+ for( j = 0 ; j < alloc_ring_buffer_num_entries ; j++ )
19758+ info->split_info[i].ring_buffers[j] = 0;
19759+ }
19760+ for( j = 0 ; j < alloc_ring_buffer_num_entries ; j++ )
19761+ {
19762+ STBIR__NEXT_PTR( info->split_info[i].ring_buffers[j], sizes.ring_buffer_length_bytes, float );
19763+ #ifdef STBIR_SIMD8
19764+ if ( ( info ) && ( effective_channels == 3 ) )
19765+ ++info->split_info[i].ring_buffers[j]; // avx in 3 channel mode needs one float at the start of the buffer
19766+ #endif
19767+ }
19768+ }
19769+#else
19770+ STBIR__NEXT_PTR( info->split_info[i].ring_buffer, sizes.ring_buffer_size + STBIR__4K_ALIAS_SKEW_SLACK, float );
19771+#endif
19772+ STBIR__NEXT_PTR( info->split_info[i].vertical_buffer, sizes.vertical_buffer_size + STBIR__4K_ALIAS_SKEW_SLACK, float );
19773+
19774+ if ( info )
19775+ {
19776+ #ifndef STBIR__SEPARATE_ALLOCATIONS
19777+ info->split_info[i].decode_buffer_alloc = info->split_info[i].decode_buffer;
19778+ info->split_info[i].ring_buffer_alloc = info->split_info[i].ring_buffer;
19779+ #endif
19780+ info->split_info[i].vertical_buffer_alloc = info->split_info[i].vertical_buffer;
19781+ }
19782+ }
19783+
19784+ // the tables are allocated into info's samplers, and the samplers are built in the locals (copied into info below)
19785+ STBIR__NEXT_PTR( info->horizontal.contributors, horizontal->contributors_size, stbir__contributors );
19786+ STBIR__NEXT_PTR( info->horizontal.coefficients, horizontal->coefficients_size, float );
19787+
19788+ if ( !copy_horizontal ) // (sharing: the vertical reads the horizontal's tables, see below)
19789+ {
19790+ STBIR__NEXT_PTR( info->vertical.contributors, vertical->contributors_size, stbir__contributors );
19791+ STBIR__NEXT_PTR( info->vertical.coefficients, vertical->coefficients_size, float );
19792+ }
19793+
19794+ if ( info )
19795+ {
19796+ horizontal->contributors = info->horizontal.contributors;
19797+ horizontal->coefficients = info->horizontal.coefficients;
19798+ vertical->contributors = info->vertical.contributors;
19799+ vertical->coefficients = info->vertical.coefficients;
19800+ }
19801+
19802+ STBIR__NEXT_PTR( info->column_slices, sizeof( stbir__column_slice ) * sizes.num_column_slices, stbir__column_slice );
19803+
19804+ if ( info )
19805+ {
19806+ STBIR_PROFILE_BUILD_START( horizontal );
19807+
19808+ stbir__calculate_filters( horizontal, user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
19809+ stbir__select_horizontal_kernels( info, horizontal, effective_channels );
19810+
19811+ info->conservative_extents = *conservative;
19812+
19813+ // pack the horizontal coeffs into the column entries the kernels walk (the vertical's aren't packed: its kernels read exactly n0..n1)
19814+ if ( horizontal->shared_period )
19815+ horizontal->coefficient_width = stbir__pack_shared_coefficients( horizontal, info->conservative_extents.n0, info->conservative_extents.n1 );
19816+ else
19817+ horizontal->coefficient_width = stbir__pack_coefficients(horizontal->num_contributors, horizontal->contributors, horizontal->coefficients, horizontal->coefficient_width, horizontal->extent_info.widest, info->conservative_extents.n0, info->conservative_extents.n1 );
19818+ horizontal->entry_floats = STBIR__COLUMN_ENTRY_FLOATS( horizontal->coefficient_width );
19819+
19820+ STBIR_MEMCPY( &info->horizontal, horizontal, sizeof( stbir__sampler ) );
19821+
19822+ // now that the contributors are final, work out exactly what each column slice decodes
19823+ info->num_column_slices = sizes.num_column_slices;
19824+ stbir__build_column_slices( info, sizes.slice_columns, sizes.slice_in_span );
19825+
19826+ STBIR_PROFILE_BUILD_END( horizontal );
19827+
19828+ STBIR_PROFILE_BUILD_START( vertical );
19829+ if ( copy_horizontal )
19830+ {
19831+ // identical samplers: the vertical reads the horizontal's packed column entries (two ints, then the coefficients)
19832+ STBIR_MEMCPY( vertical, horizontal, sizeof( stbir__sampler ) );
19833+ vertical->coefficients += 2;
19834+ vertical->coefficient_width = horizontal->entry_floats;
19835+ }
19836+ else
19837+ stbir__calculate_filters( vertical, user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
19838+ STBIR_MEMCPY( &info->vertical, vertical, sizeof( stbir__sampler ) );
19839+ STBIR_PROFILE_BUILD_END( vertical );
19840+
19841+ // the horizontal's contributors have been read for the last time (the slice table), so fill in the column entries'
19842+ // advance and width (an advance that doesn't fit fails the build)
19843+ if ( !stbir__fill_column_entries( &info->horizontal, effective_channels ) )
19844+ {
19845+ stbir__free_internal_mem( info );
19846+ return 0;
19847+ }
19848+
19849+ stbir__setup_split_grid( info, split_cols, split_rows );
19850+
19851+ // now we know precisely how many entries we need
19852+ info->ring_buffer_num_entries = info->vertical.extent_info.widest;
19853+
19854+ // we never need more ring buffer entries than the scanlines we're outputting
19855+ if ( ( !info->vertical.is_gather ) && ( info->ring_buffer_num_entries > conservative_split_output_size ) )
19856+ info->ring_buffer_num_entries = conservative_split_output_size;
19857+ STBIR_ASSERT( info->ring_buffer_num_entries <= info->alloc_ring_buffer_num_entries );
19858+ }
19859+ #undef STBIR__NEXT_PTR
19860+
19861+ // is this the first time through loop?
19862+ if ( info == 0 )
19863+ {
19864+ alloced_total = ( 15 + (size_t)advance_mem );
19865+ alloced = STBIR_MALLOC( alloced_total, user_data );
19866+ if ( alloced == 0 )
19867+ return 0;
19868+ }
19869+ else
19870+ return info; // success
19871+ }
19872+}
19873+
19874+// 4K aliasing avoidance: each internal buffer has a page of slack and is placed half a page from the buffer
19875+// that streams into it - input -> decode -> ring -> vertical buffer (horizontal first),
19876+// input -> ring -> decode -> vertical buffer (vertical first).
19877+static size_t stbir__4K_alias_place( void * alloc, size_t want )
19878+{
19879+ size_t have = ( (size_t)alloc ) & 4095;
19880+ return ( ( want - have ) & 4095 ) & ~(size_t)15; // stays 16 byte aligned, within the slack
19881+}
19882+
19883+static void stbir__4K_alias_skew_buffers( stbir__info const * info, stbir__per_split_info * split_info )
19884+{
19885+ size_t first = ( ( (size_t)info->input_data ) + 2048 ) & 4095; // the first internal buffer: half a page from the input row
19886+ size_t second = ( first + 2048 ) & 4095; // and the next one, back at the input's offset
19887+ size_t vertical_dest;
19888+
19889+ if ( info->vertical_first )
19890+ {
19891+ // decode: input -> ring. vertical: ring -> decode_buffer. horizontal: decode_buffer -> vertical_buffer. encode: vertical_buffer -> output
19892+ #ifndef STBIR__SEPARATE_ALLOCATIONS
19893+ split_info->ring_buffer = (float*) ( (char*)split_info->ring_buffer_alloc + stbir__4K_alias_place( split_info->ring_buffer_alloc, first ) );
19894+ #endif
19895+ vertical_dest = ( second + (size_t)stbir__4K_vertical_dest_shift ) & 4095;
19896+ split_info->decode_buffer = (float*) ( (char*)split_info->decode_buffer_alloc + stbir__4K_alias_place( split_info->decode_buffer_alloc, vertical_dest ) );
19897+ split_info->vertical_buffer = (float*) ( (char*)split_info->vertical_buffer_alloc + stbir__4K_alias_place( split_info->vertical_buffer_alloc, ( vertical_dest + 2048 ) & 4095 ) );
19898+ }
19899+ else
19900+ {
19901+ // decode: input -> decode_buffer. horizontal: decode_buffer -> ring. vertical: ring -> vertical_buffer. encode: vertical_buffer -> output
19902+ split_info->decode_buffer = (float*) ( (char*)split_info->decode_buffer_alloc + stbir__4K_alias_place( split_info->decode_buffer_alloc, first ) );
19903+ #ifndef STBIR__SEPARATE_ALLOCATIONS
19904+ split_info->ring_buffer = (float*) ( (char*)split_info->ring_buffer_alloc + stbir__4K_alias_place( split_info->ring_buffer_alloc, second ) );
19905+ #endif
19906+ vertical_dest = ( first + (size_t)stbir__4K_vertical_dest_shift ) & 4095;
19907+ split_info->vertical_buffer = (float*) ( (char*)split_info->vertical_buffer_alloc + stbir__4K_alias_place( split_info->vertical_buffer_alloc, vertical_dest ) );
19908+ }
19909+}
19910+
19911+static int stbir__perform_resize( stbir__info const * info, int split_start, int split_count )
19912+{
19913+ stbir__per_split_info * split_info = info->split_info + split_start;
19914+ int b, i, can_merge;
19915+
19916+ for( i = 0 ; i < split_count ; i++ )
19917+ stbir__4K_alias_skew_buffers( info, split_info + i );
19918+
19919+ STBIR_PROFILE_CLEAR_EXTRAS();
19920+
19921+ STBIR_PROFILE_FIRST_START( looping );
19922+ // run the whole vertical loop once per column slice, so each pass only touches a cache sized slice of the buffers.
19923+ // consecutive splits over the same slices are adjacent row bands and run as one band (one warm-up per slice) - except
19924+ // when a scatter's ring was cut down to a band's rows: an input row can touch more output rows than that across
19925+ // two bands, so those run one at a time
19926+ can_merge = info->vertical.is_gather || ( info->ring_buffer_num_entries >= info->vertical.extent_info.widest );
19927+ i = 0;
19928+ while ( i < split_count )
19929+ {
19930+ int run = 1;
19931+ while ( can_merge && ( i + run < split_count ) && ( split_info[i+run].first_column_slice == split_info[i].first_column_slice ) && ( split_info[i+run].num_column_slices == split_info[i].num_column_slices ) )
19932+ ++run;
19933+ STBIR_PROFILE_NEST_IN_FIRST( split_info + i );
19934+ for( b = 0 ; b < split_info[i].num_column_slices ; b++ )
19935+ {
19936+ stbir__column_slice const * slice = info->column_slices + split_info[i].first_column_slice + b;
19937+ if (info->vertical.is_gather)
19938+ stbir__vertical_gather_loop( info, split_info + i, run, slice );
19939+ else
19940+ stbir__vertical_scatter_loop( info, split_info + i, run, slice );
19941+ }
19942+ i += run;
19943+ }
19944+ STBIR_PROFILE_END( looping );
19945+
19946+ return 1;
19947+}
19948+
19949+static void stbir__update_info_from_resize( stbir__info * info, STBIR_RESIZE * resize )
19950+{
19951+ static stbir__decode_pixels_func * decode_simple[STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
19952+ {
19953+ /* 1ch-4ch */ stbir__decode_uint8_srgb, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear,
19954+ };
19955+
19956+ static stbir__decode_pixels_func * decode_alphas[STBIRI_AR-STBIRI_RGBA+1][STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
19957+ {
19958+ { /* RGBA */ stbir__decode_uint8_srgb4_linearalpha, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear },
19959+ { /* BGRA */ stbir__decode_uint8_srgb4_linearalpha_BGRA, stbir__decode_uint8_srgb_BGRA, 0, stbir__decode_float_linear_BGRA, stbir__decode_half_float_linear_BGRA },
19960+ { /* ARGB */ stbir__decode_uint8_srgb4_linearalpha_ARGB, stbir__decode_uint8_srgb_ARGB, 0, stbir__decode_float_linear_ARGB, stbir__decode_half_float_linear_ARGB },
19961+ { /* ABGR */ stbir__decode_uint8_srgb4_linearalpha_ABGR, stbir__decode_uint8_srgb_ABGR, 0, stbir__decode_float_linear_ABGR, stbir__decode_half_float_linear_ABGR },
19962+ { /* RA */ stbir__decode_uint8_srgb2_linearalpha, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear },
19963+ { /* AR */ stbir__decode_uint8_srgb2_linearalpha_AR, stbir__decode_uint8_srgb_AR, 0, stbir__decode_float_linear_AR, stbir__decode_half_float_linear_AR },
19964+ };
19965+
19966+ static stbir__decode_pixels_func * decode_simple_scaled_or_not[2][2]=
19967+ {
19968+ { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear },
19969+ };
19970+
19971+ static stbir__decode_pixels_func * decode_alphas_scaled_or_not[STBIRI_AR-STBIRI_RGBA+1][2][2]=
19972+ {
19973+ { /* RGBA */ { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear } },
19974+ { /* BGRA */ { stbir__decode_uint8_linear_scaled_BGRA, stbir__decode_uint8_linear_BGRA }, { stbir__decode_uint16_linear_scaled_BGRA, stbir__decode_uint16_linear_BGRA } },
19975+ { /* ARGB */ { stbir__decode_uint8_linear_scaled_ARGB, stbir__decode_uint8_linear_ARGB }, { stbir__decode_uint16_linear_scaled_ARGB, stbir__decode_uint16_linear_ARGB } },
19976+ { /* ABGR */ { stbir__decode_uint8_linear_scaled_ABGR, stbir__decode_uint8_linear_ABGR }, { stbir__decode_uint16_linear_scaled_ABGR, stbir__decode_uint16_linear_ABGR } },
19977+ { /* RA */ { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear } },
19978+ { /* AR */ { stbir__decode_uint8_linear_scaled_AR, stbir__decode_uint8_linear_AR }, { stbir__decode_uint16_linear_scaled_AR, stbir__decode_uint16_linear_AR } }
19979+ };
19980+
19981+ static stbir__encode_pixels_func * encode_simple[STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
19982+ {
19983+ /* 1ch-4ch */ stbir__encode_uint8_srgb, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear,
19984+ };
19985+
19986+ static stbir__encode_pixels_func * encode_alphas[STBIRI_AR-STBIRI_RGBA+1][STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
19987+ {
19988+ { /* RGBA */ stbir__encode_uint8_srgb4_linearalpha, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear },
19989+ { /* BGRA */ stbir__encode_uint8_srgb4_linearalpha_BGRA, stbir__encode_uint8_srgb_BGRA, 0, stbir__encode_float_linear_BGRA, stbir__encode_half_float_linear_BGRA },
19990+ { /* ARGB */ stbir__encode_uint8_srgb4_linearalpha_ARGB, stbir__encode_uint8_srgb_ARGB, 0, stbir__encode_float_linear_ARGB, stbir__encode_half_float_linear_ARGB },
19991+ { /* ABGR */ stbir__encode_uint8_srgb4_linearalpha_ABGR, stbir__encode_uint8_srgb_ABGR, 0, stbir__encode_float_linear_ABGR, stbir__encode_half_float_linear_ABGR },
19992+ { /* RA */ stbir__encode_uint8_srgb2_linearalpha, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear },
19993+ { /* AR */ stbir__encode_uint8_srgb2_linearalpha_AR, stbir__encode_uint8_srgb_AR, 0, stbir__encode_float_linear_AR, stbir__encode_half_float_linear_AR }
19994+ };
19995+
19996+ static stbir__encode_pixels_func * encode_simple_scaled_or_not[2][2]=
19997+ {
19998+ { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear },
19999+ };
20000+
20001+ static stbir__encode_pixels_func * encode_alphas_scaled_or_not[STBIRI_AR-STBIRI_RGBA+1][2][2]=
20002+ {
20003+ { /* RGBA */ { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear } },
20004+ { /* BGRA */ { stbir__encode_uint8_linear_scaled_BGRA, stbir__encode_uint8_linear_BGRA }, { stbir__encode_uint16_linear_scaled_BGRA, stbir__encode_uint16_linear_BGRA } },
20005+ { /* ARGB */ { stbir__encode_uint8_linear_scaled_ARGB, stbir__encode_uint8_linear_ARGB }, { stbir__encode_uint16_linear_scaled_ARGB, stbir__encode_uint16_linear_ARGB } },
20006+ { /* ABGR */ { stbir__encode_uint8_linear_scaled_ABGR, stbir__encode_uint8_linear_ABGR }, { stbir__encode_uint16_linear_scaled_ABGR, stbir__encode_uint16_linear_ABGR } },
20007+ { /* RA */ { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear } },
20008+ { /* AR */ { stbir__encode_uint8_linear_scaled_AR, stbir__encode_uint8_linear_AR }, { stbir__encode_uint16_linear_scaled_AR, stbir__encode_uint16_linear_AR } }
20009+ };
20010+
20011+ stbir__decode_pixels_func * decode_pixels = 0;
20012+ stbir__encode_pixels_func * encode_pixels = 0;
20013+ stbir_datatype input_type, output_type;
20014+ int non_scaled;
20015+
20016+ input_type = resize->input_data_type;
20017+ output_type = resize->output_data_type;
20018+ info->input_data = resize->input_pixels;
20019+ info->input_stride_bytes = resize->input_stride_in_bytes;
20020+ info->output_stride_bytes = resize->output_stride_in_bytes;
20021+
20022+ // if we're completely point sampling, then we can turn off SRGB
20023+ if ( ( info->horizontal.filter_enum == STBIR_FILTER_POINT_SAMPLE ) && ( info->vertical.filter_enum == STBIR_FILTER_POINT_SAMPLE ) )
20024+ {
20025+ if ( ( ( input_type == STBIR_TYPE_UINT8_SRGB ) || ( input_type == STBIR_TYPE_UINT8_SRGB_ALPHA ) ) &&
20026+ ( ( output_type == STBIR_TYPE_UINT8_SRGB ) || ( output_type == STBIR_TYPE_UINT8_SRGB_ALPHA ) ) )
20027+ {
20028+ input_type = STBIR_TYPE_UINT8;
20029+ output_type = STBIR_TYPE_UINT8;
20030+ }
20031+ }
20032+
20033+ // recalc the output and input strides (zero means packed - the whole image's width, the sub rect is a region of it)
20034+ if ( info->input_stride_bytes == 0 )
20035+ info->input_stride_bytes = (stbir_int64)info->channels * info->horizontal.scale_info.input_full_size * stbir__type_size[input_type];
20036+
20037+ if ( info->output_stride_bytes == 0 )
20038+ info->output_stride_bytes = (stbir_int64)info->channels * resize->output_w * stbir__type_size[output_type];
20039+
20040+ // calc offset
20041+ info->output_data = ( (char*) resize->output_pixels ) + (ptrdiff_t)( info->offset_y * info->output_stride_bytes ) + ( (size_t) info->offset_x * (size_t) info->channels * (size_t) stbir__type_size[output_type] );
20042+
20043+ info->in_pixels_cb = resize->input_cb;
20044+ info->user_data = resize->user_data;
20045+ info->out_pixels_cb = resize->output_cb;
20046+
20047+ // check if we can run unscaled - 0-255.0/0-65535.0 instead of 0-1.0 (which is a tiny bit faster when doing linear 8->8 or 16->16)
20048+ // don't short circuit when alpha weighting (get everything to 0-1.0 as usual)
20049+ non_scaled = 0;
20050+ if ( ( !info->alpha_weight ) && ( !info->alpha_unweight ) )
20051+ if ( ( ( input_type == STBIR_TYPE_UINT8 ) && ( output_type == STBIR_TYPE_UINT8 ) ) || ( ( input_type == STBIR_TYPE_UINT16 ) && ( output_type == STBIR_TYPE_UINT16 ) ) )
20052+ non_scaled = 1;
20053+
20054+ // setup the input format converters
20055+ if ( ( input_type == STBIR_TYPE_UINT8 ) || ( input_type == STBIR_TYPE_UINT16 ) )
20056+ {
20057+ if ( info->input_pixel_layout_internal <= STBIRI_4CHANNEL )
20058+ decode_pixels = decode_simple_scaled_or_not[ input_type == STBIR_TYPE_UINT16 ][ non_scaled ];
20059+ else
20060+ decode_pixels = decode_alphas_scaled_or_not[ ( info->input_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ input_type == STBIR_TYPE_UINT16 ][ non_scaled ];
20061+ }
20062+ else
20063+ {
20064+ if ( info->input_pixel_layout_internal <= STBIRI_4CHANNEL )
20065+ decode_pixels = decode_simple[ input_type - STBIR_TYPE_UINT8_SRGB ];
20066+ else
20067+ decode_pixels = decode_alphas[ ( info->input_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ input_type - STBIR_TYPE_UINT8_SRGB ];
20068+ }
20069+
20070+ // setup the output format converters
20071+ if ( ( output_type == STBIR_TYPE_UINT8 ) || ( output_type == STBIR_TYPE_UINT16 ) )
20072+ {
20073+ if ( info->output_pixel_layout_internal <= STBIRI_4CHANNEL )
20074+ encode_pixels = encode_simple_scaled_or_not[ output_type == STBIR_TYPE_UINT16 ][ non_scaled ];
20075+ else
20076+ encode_pixels = encode_alphas_scaled_or_not[ ( info->output_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ output_type == STBIR_TYPE_UINT16 ][ non_scaled ];
20077+ }
20078+ else
20079+ {
20080+ if ( info->output_pixel_layout_internal <= STBIRI_4CHANNEL )
20081+ encode_pixels = encode_simple[ output_type - STBIR_TYPE_UINT8_SRGB ];
20082+ else
20083+ encode_pixels = encode_alphas[ ( info->output_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ output_type - STBIR_TYPE_UINT8_SRGB ];
20084+ }
20085+
20086+ info->input_type = input_type;
20087+ info->output_type = output_type;
20088+ info->decode_pixels = decode_pixels;
20089+ info->encode_pixels = encode_pixels;
20090+
20091+ // the input callback's temp buffer sits inside the decode buffer, pushed forward so the decoder reads ahead of
20092+ // its writes - except plain float, which is a no-op decode when the temp is exactly the decode target
20093+ info->input_callback_offset_bytes = ( decode_pixels == stbir__decode_float_linear ) ? 0 : ( sizeof(float) * STBIR_INPUT_CALLBACK_PADDING );
20094+}
20095+
20096+static void stbir__clip( int * outx, int * outsubw, int outw, stbir_real * u0, stbir_real * u1 )
20097+{
20098+ stbir_real per, adj;
20099+ int over;
20100+
20101+ // do left/top edge
20102+ if ( *outx < 0 )
20103+ {
20104+ per = ( (stbir_real)*outx ) / ( (stbir_real)*outsubw ); // is negative
20105+ adj = per * ( *u1 - *u0 );
20106+ *u0 -= adj; // increases u0
20107+ *outsubw += *outx; // and drop the clipped off pixels from the output count
20108+ *outx = 0;
20109+ }
20110+
20111+ // do right/bot edge
20112+ over = outw - ( *outx + *outsubw );
20113+ if ( over < 0 )
20114+ {
20115+ per = ( (stbir_real)over ) / ( (stbir_real)*outsubw ); // is negative
20116+ adj = per * ( *u1 - *u0 );
20117+ *u1 += adj; // decrease u1
20118+ *outsubw = outw - *outx;
20119+ }
20120+}
20121+
20122+static stbir_uint64 stbir__gcd( stbir_uint64 a, stbir_uint64 b )
20123+{
20124+ while ( b )
20125+ {
20126+ stbir_uint64 remainder = a % b;
20127+ a = b;
20128+ b = remainder;
20129+ }
20130+ return a;
20131+}
20132+
20133+// The simplest fraction within tolerance of f (f>0): its continued fraction's first convergent that is, in 31 bits (or 0).
20134+static int stbir__real_to_fraction( stbir_real f, stbir_real tolerance, stbir_uint32 * out_numer, stbir_uint32 * out_denom )
20135+{
20136+ stbir_real rem = f;
20137+ stbir_uint64 numer = 1, denom = 0, numer_last = 0, denom_last = 1; // the convergents numer / denom
20138+ int i;
20139+
20140+ if ( f <= (stbir_real) 0.0 )
20141+ return 0;
20142+ for( i = 0 ; i < (int) sizeof( stbir_real ) * 8 ; i++ ) // (a term is at least a bit of f)
20143+ {
20144+ stbir_uint64 term = (stbir_uint64) rem; // the floor (rem is positive)
20145+ stbir_uint64 next;
20146+ stbir_real frac, err;
20147+
20148+ if ( ( term > 0x7fffffff ) || ( ( term * numer + numer_last ) > 0x7fffffff ) || ( ( term * denom + denom_last ) > 0x7fffffff ) )
20149+ return 0;
20150+ next = term * numer + numer_last; numer_last = numer; numer = next;
20151+ next = term * denom + denom_last; denom_last = denom; denom = next;
20152+
20153+ err = (stbir_real) numer / (stbir_real) denom - f;
20154+ if ( err < (stbir_real) 0.0 )
20155+ err = -err;
20156+ if ( err <= tolerance )
20157+ {
20158+ *out_numer = (stbir_uint32) numer;
20159+ *out_denom = (stbir_uint32) denom;
20160+ return 1;
20161+ }
20162+
20163+ frac = rem - (stbir_real) term;
20164+ if ( frac <= (stbir_real) 0.0 )
20165+ return 0; // (the fraction is exactly f, and still not within tolerance - can't happen for a tolerance >= 0)
20166+ rem = (stbir_real) 1.0 / frac;
20167+ }
20168+ return 0;
20169+}
20170+
20171+// try to find approximation that stays under the drift value: the far edge of the output moves by less than
20172+// 1 / STBIR__POLYPHASE_DRIFT_DIV of an output pixel
20173+#define STBIR__POLYPHASE_DRIFT_DIV 8192
20174+static void stbir__approximate_polyphase( stbir_uint64 numer, stbir_uint64 denom, stbir_uint64 outputs, stbir_uint64 * out_numer, stbir_uint64 * out_denom, int * out_drift_in_pixels )
20175+{
20176+ stbir_uint64 top = numer, bot = denom;
20177+ stbir_uint64 p = 1, q = 0, p_last = 0, q_last = 1;
20178+
20179+ while ( bot )
20180+ {
20181+ stbir_uint64 est = top / bot;
20182+ stbir_uint64 remainder = top % bot;
20183+ stbir_uint64 next, pd, qn, diff;
20184+
20185+ top = bot;
20186+ bot = remainder;
20187+
20188+ // the next convergent p / q
20189+ next = est * p + p_last;
20190+ p_last = p;
20191+ p = next;
20192+
20193+ next = est * q + q_last;
20194+ q_last = q;
20195+ q = next;
20196+
20197+ // no simpler fraction is close enough (the convergents only grow from here)
20198+ if ( p >= outputs )
20199+ return;
20200+
20201+ // its drift at the far edge, in output pixels: outputs x | p D - q N | / ( q N )
20202+ pd = p * denom;
20203+ qn = q * numer;
20204+ diff = ( pd > qn ) ? pd - qn : qn - pd;
20205+ if ( diff <= qn / ( outputs * STBIR__POLYPHASE_DRIFT_DIV ) )
20206+ {
20207+ // that drift at the far edge in input pixels (times denom / numer), rounded up - for the conservative range
20208+ stbir_real drift_in = (stbir_real)outputs * (stbir_real)diff * (stbir_real)denom / ( (stbir_real)qn * (stbir_real)numer );
20209+ *out_numer = p;
20210+ *out_denom = q;
20211+ *out_drift_in_pixels = (int)drift_in + 1;
20212+ return;
20213+ }
20214+ }
20215+}
20216+
20217+// The scale as a fraction: output_sub / ( input_full x ( s1 - s0 ) ), with s1 - s0 = ( n1 - n0 ) / d - exact if possible.
20218+// Returns 0 when there is no fraction that fits in int.
20219+static int stbir__scale_to_rational( int output_sub_range, int input_full_range, stbir_real input_s, int s_n0, int s_n1, int s_d, stbir_uint32 * out_numer, stbir_uint32 * out_denom, int * out_approximated_drift )
20220+{
20221+ stbir_uint64 numer, denom, g;
20222+ stbir_real scale, err;
20223+
20224+ *out_numer = 0; // (not a fraction: the fields are read before the flag in places, so never leave them unset)
20225+ *out_denom = 0;
20226+ *out_approximated_drift = 0;
20227+
20228+ if ( ( s_d > 0 ) && ( s_n1 > s_n0 ) )
20229+ {
20230+ numer = (stbir_uint64)output_sub_range * (stbir_uint64)s_d;
20231+ denom = (stbir_uint64)input_full_range * (stbir_uint64)( s_n1 - s_n0 );
20232+ g = stbir__gcd( numer, denom );
20233+ numer /= g; denom /= g;
20234+ if ( ( numer <= 0x7fffffff ) && ( denom <= 0x7fffffff ) )
20235+ {
20236+ if ( numer >= (stbir_uint64)output_sub_range )
20237+ stbir__approximate_polyphase( numer, denom, (stbir_uint64)output_sub_range, &numer, &denom, out_approximated_drift );
20238+ *out_numer = (stbir_uint32) numer;
20239+ *out_denom = (stbir_uint32) denom;
20240+ return 1;
20241+ }
20242+ }
20243+
20244+ scale = (stbir_real)output_sub_range / ( (stbir_real)input_full_range * input_s );
20245+ if ( !stbir__real_to_fraction( scale, scale / ( (stbir_real)output_sub_range * (stbir_real)STBIR__POLYPHASE_DRIFT_DIV ), out_numer, out_denom ) )
20246+ return 0;
20247+ err = (stbir_real)*out_numer / (stbir_real)*out_denom - scale;
20248+ if ( err < (stbir_real) 0.0 )
20249+ err = -err;
20250+ if ( err > (stbir_real) 0.0 )
20251+ *out_approximated_drift = (int)( (stbir_real)output_sub_range * err / ( scale * scale ) ) + 1; // the far edge's drift: output pixels x ( err / scale ), in input pixels ( / scale )
20252+ return 1;
20253+}
20254+
20255+// clip the input area to the image (and adjust the output area), the counterpart of stbir__clip: the output pixels whose
20256+// centers land outside the image are dropped from the ends, so the ones left sample exactly where they did
20257+static void stbir__clip_input( int * outx, int * outsubw, stbir_real * u0, stbir_real * u1 )
20258+{
20259+ stbir_real per_pixel = ( *u1 - *u0 ) / ( (stbir_real)*outsubw ); // input area per output pixel
20260+ int drop;
20261+
20262+ // do left/top edge
20263+ if ( *u0 < 0 )
20264+ {
20265+ drop = stbir__ceil_to_int( -*u0 / per_pixel - (stbir_real)0.5 ); // pixels centered before the image (one right on the edge stays)
20266+ if ( drop > *outsubw )
20267+ drop = *outsubw;
20268+ if ( drop > 0 )
20269+ {
20270+ *u0 += (stbir_real)drop * per_pixel; // increases u0
20271+ *outx += drop;
20272+ *outsubw -= drop;
20273+ }
20274+ }
20275+
20276+ // do right/bot edge
20277+ if ( *u1 > 1 )
20278+ {
20279+ drop = stbir__ceil_to_int( ( *u1 - 1 ) / per_pixel - (stbir_real)0.5 ); // pixels centered past the image
20280+ if ( drop > *outsubw )
20281+ drop = *outsubw;
20282+ if ( drop > 0 )
20283+ {
20284+ *u1 -= (stbir_real)drop * per_pixel; // decreases u1
20285+ *outsubw -= drop;
20286+ }
20287+ }
20288+}
20289+
20290+static int stbir__calculate_region_transform( stbir__scale_info * scale_info, int output_full_range, int * output_offset, int output_sub_range, int input_full_range, stbir_real input_s0, stbir_real input_s1, int s_n0, int s_n1, int s_d )
20291+{
20292+ stbir_real output_range, input_range, output_s, input_s, input_pixels, ratio, unclipped_input_s;
20293+ int unclipped_output_sub_range = output_sub_range;
20294+ int unclipped_output_offset = *output_offset;
20295+
20296+ input_s = input_s1 - input_s0;
20297+ unclipped_input_s = input_s;
20298+
20299+ // null area (a negative size is an empty one)
20300+ if ( ( output_full_range <= 0 ) || ( input_full_range <= 0 ) ||
20301+ ( output_sub_range <= 0 ) || ( input_s <= (stbir_real)0.0 ) )
20302+ return 0;
20303+
20304+ // are either of the ranges completely out of bounds?
20305+ if ( ( *output_offset >= output_full_range ) || ( ( *output_offset + output_sub_range ) <= 0 ) || ( input_s0 >= (stbir_real)1.0 ) || ( input_s1 <= (stbir_real)0.0 ) )
20306+ return 0;
20307+
20308+ output_range = (stbir_real)output_full_range;
20309+ input_range = (stbir_real)input_full_range;
20310+
20311+ output_s = ( (stbir_real)output_sub_range) / output_range;
20312+
20313+ // figure out the scaling to use
20314+ ratio = output_s / input_s;
20315+
20316+ // save scale before clipping: the output pixels over the input pixels the region spans, and its inverse as the same
20317+ // two numbers divided the other way - one rounding each, and since a division is monotonic the two agree about 1
20318+ // (a scale that rounds to exactly 1 can't have an inverse over 1, which would put a ratio over 1 to the kernels)
20319+ input_pixels = input_range * input_s;
20320+ scale_info->scale = (stbir_real)output_sub_range / input_pixels;
20321+ scale_info->inv_scale = input_pixels / (stbir_real)output_sub_range;
20322+
20323+ // clip output area to left/right output edges (and adjust input area)
20324+ stbir__clip( output_offset, &output_sub_range, output_full_range, &input_s0, &input_s1 );
20325+
20326+ // clip input area to the image (and adjust output area)
20327+ stbir__clip_input( output_offset, &output_sub_range, &input_s0, &input_s1 );
20328+
20329+ // recalc input area
20330+ input_s = input_s1 - input_s0;
20331+
20332+ // after clipping do we have zero input area?
20333+ if ( ( input_s <= (stbir_real)0.0 ) || ( output_sub_range <= 0 ) )
20334+ return 0;
20335+
20336+ // calculate and store the starting source offsets in output pixel space
20337+ scale_info->pixel_shift = input_s0 * ratio * output_range;
20338+
20339+ scale_info->scale_is_rational = stbir__scale_to_rational( unclipped_output_sub_range, input_full_range, unclipped_input_s, s_n0, s_n1, s_d, &scale_info->scale_numerator, &scale_info->scale_denominator, &scale_info->approximated_drift );
20340+
20341+ // an exact fraction gives the scale itself - correctly rounded, so an exact ratio's filter widths ( ceil( 2 support / scale ) )
20342+ // don't pick up an extra tap from the region math landing an ulp under the integer, and a 1:1 is exactly 1 (the up / down
20343+ // decision). an approximated fraction leaves the real scale: it is what the kernels see, and the decision has to agree with it
20344+ if ( ( scale_info->scale_is_rational ) && ( !scale_info->approximated_drift ) )
20345+ {
20346+ scale_info->scale = (stbir_real) scale_info->scale_numerator / (stbir_real) scale_info->scale_denominator;
20347+ scale_info->inv_scale = (stbir_real) scale_info->scale_denominator / (stbir_real) scale_info->scale_numerator;
20348+ }
20349+
20350+ // exact tap positions when the shift is an integer ( n0 x output_sub / ( n1 - n0 ), plus the clip ) and the fraction wasn't approximated.
20351+ // the numerators are ( 2o + 1 + 2 shift ) D and ( 2i + 1 ) N: with sizes up to 2^30 (the resizer's limit) and the
20352+ // fraction's terms below 2^30 they stay below 2^62
20353+ scale_info->positions_exact = 0;
20354+ scale_info->shift_pixels = 0;
20355+ if ( ( scale_info->scale_is_rational ) && ( !scale_info->approximated_drift ) && ( s_d > 0 ) && ( ( ( (stbir_int64)s_n0 * unclipped_output_sub_range ) % ( s_n1 - s_n0 ) ) == 0 ) &&
20356+ ( scale_info->scale_numerator < ( 1 << 30 ) ) && ( scale_info->scale_denominator < ( 1 << 30 ) ) )
20357+ {
20358+ stbir_int64 shift = ( (stbir_int64)s_n0 * unclipped_output_sub_range ) / ( s_n1 - s_n0 ) + ( *output_offset - unclipped_output_offset );
20359+ if ( ( shift > -( 1 << 30 ) ) && ( shift < ( 1 << 30 ) ) ) // the shift is in output pixels too (an output tile far past the image's own size)
20360+ {
20361+ scale_info->shift_pixels = (int) shift;
20362+ scale_info->tap_scale_in = (stbir_real) 1.0 / ( (stbir_real) 2.0 * (stbir_real) scale_info->scale_numerator );
20363+ scale_info->tap_scale_out = (stbir_real) 1.0 / ( (stbir_real) 2.0 * (stbir_real) scale_info->scale_denominator );
20364+ scale_info->pixel_shift = (stbir_real) shift;
20365+ scale_info->positions_exact = 1;
20366+ }
20367+ }
20368+
20369+ scale_info->input_full_size = input_full_range;
20370+ scale_info->output_sub_size = output_sub_range;
20371+
20372+ return 1;
20373+}
20374+
20375+
20376+static void stbir__init_and_set_layout( STBIR_RESIZE * resize, stbir_pixel_layout pixel_layout, stbir_datatype data_type )
20377+{
20378+ resize->input_cb = 0;
20379+ resize->output_cb = 0;
20380+ resize->user_data = resize;
20381+ resize->samplers = 0;
20382+ resize->called_alloc = 0;
20383+ resize->horizontal_filter = STBIR_FILTER_DEFAULT;
20384+ resize->horizontal_filter_kernel = 0; resize->horizontal_filter_support = 0; resize->horizontal_filter_kernel_new = 0; resize->horizontal_filter_support_new = 0; resize->horizontal_filter_radius = 0;
20385+ resize->vertical_filter = STBIR_FILTER_DEFAULT;
20386+ resize->vertical_filter_kernel = 0; resize->vertical_filter_support = 0; resize->vertical_filter_kernel_new = 0; resize->vertical_filter_support_new = 0; resize->vertical_filter_radius = 0;
20387+ resize->horizontal_edge = STBIR_EDGE_CLAMP;
20388+ resize->vertical_edge = STBIR_EDGE_CLAMP;
20389+ resize->input_s0 = 0; resize->input_t0 = 0; resize->input_s1 = 1; resize->input_t1 = 1;
20390+ resize->input_s_n0 = 0; resize->input_s_n1 = 1; resize->input_s_d = 1;
20391+ resize->input_t_n0 = 0; resize->input_t_n1 = 1; resize->input_t_d = 1;
20392+ resize->output_subx = 0; resize->output_suby = 0; resize->output_subw = resize->output_w; resize->output_subh = resize->output_h;
20393+ resize->input_data_type = data_type;
20394+ resize->output_data_type = data_type;
20395+ resize->input_pixel_layout_public = pixel_layout;
20396+ resize->output_pixel_layout_public = pixel_layout;
20397+ resize->needs_rebuild = 1;
20398+}
20399+
20400+STBIRDEF void stbir_resize_init( STBIR_RESIZE * resize,
20401+ const void *input_pixels, int input_w, int input_h, stbir_int64 input_stride_in_bytes, // stride can be zero
20402+ void *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes, // stride can be zero
20403+ stbir_pixel_layout pixel_layout, stbir_datatype data_type )
20404+{
20405+ resize->input_pixels = input_pixels;
20406+ resize->input_w = input_w;
20407+ resize->input_h = input_h;
20408+ resize->input_stride_in_bytes = input_stride_in_bytes;
20409+ resize->output_pixels = output_pixels;
20410+ resize->output_w = output_w;
20411+ resize->output_h = output_h;
20412+ resize->output_stride_in_bytes = output_stride_in_bytes;
20413+ resize->fast_alpha = 0;
20414+
20415+ stbir__init_and_set_layout( resize, pixel_layout, data_type );
20416+}
20417+
20418+// You can update parameters any time after resize_init
20419+STBIRDEF void stbir_set_datatypes( STBIR_RESIZE * resize, stbir_datatype input_type, stbir_datatype output_type ) // by default, datatype from resize_init
20420+{
20421+ resize->input_data_type = input_type;
20422+ resize->output_data_type = output_type;
20423+ if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
20424+ stbir__update_info_from_resize( resize->samplers, resize );
20425+}
20426+
20427+STBIRDEF void stbir_set_pixel_callbacks( STBIR_RESIZE * resize, stbir_input_callback * input_cb, stbir_output_callback * output_cb ) // no callbacks by default
20428+{
20429+ resize->input_cb = input_cb;
20430+ resize->output_cb = output_cb;
20431+
20432+ if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
20433+ {
20434+ resize->samplers->in_pixels_cb = input_cb;
20435+ resize->samplers->out_pixels_cb = output_cb;
20436+ }
20437+}
20438+
20439+STBIRDEF void stbir_set_user_data( STBIR_RESIZE * resize, void * user_data ) // pass back STBIR_RESIZE* by default
20440+{
20441+ resize->user_data = user_data;
20442+ if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
20443+ resize->samplers->user_data = user_data;
20444+}
20445+
20446+STBIRDEF void stbir_set_buffer_ptrs( STBIR_RESIZE * resize, const void * input_pixels, stbir_int64 input_stride_in_bytes, void * output_pixels, stbir_int64 output_stride_in_bytes )
20447+{
20448+ resize->input_pixels = input_pixels;
20449+ resize->input_stride_in_bytes = input_stride_in_bytes;
20450+ resize->output_pixels = output_pixels;
20451+ resize->output_stride_in_bytes = output_stride_in_bytes;
20452+ if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
20453+ stbir__update_info_from_resize( resize->samplers, resize );
20454+}
20455+
20456+
20457+STBIRDEF int stbir_set_edgemodes( STBIR_RESIZE * resize, stbir_edge horizontal_edge, stbir_edge vertical_edge ) // CLAMP by default
20458+{
20459+ resize->horizontal_edge = horizontal_edge;
20460+ resize->vertical_edge = vertical_edge;
20461+ resize->needs_rebuild = 1;
20462+ return 1;
20463+}
20464+
20465+STBIRDEF int stbir_set_filters( STBIR_RESIZE * resize, stbir_filter horizontal_filter, stbir_filter vertical_filter ) // STBIR_DEFAULT_FILTER_UPSAMPLE/DOWNSAMPLE by default
20466+{
20467+ resize->horizontal_filter = horizontal_filter;
20468+ resize->vertical_filter = vertical_filter;
20469+ resize->needs_rebuild = 1;
20470+ return 1;
20471+}
20472+
20473+STBIRDEF int stbir_set_filter_callbacks_new( STBIR_RESIZE * resize, stbir__kernel_callback_new * horizontal_filter, stbir__support_callback_new * horizontal_support, stbir__radius_callback_new * horizontal_radius, stbir__kernel_callback_new * vertical_filter, stbir__support_callback_new * vertical_support, stbir__radius_callback_new * vertical_radius )
20474+{
20475+ // a filter's extent is given one way: a support (filter pixels) or a radius (input pixels), not both
20476+ if ( ( horizontal_support && horizontal_radius ) || ( vertical_support && vertical_radius ) )
20477+ return 0;
20478+ resize->horizontal_filter_kernel_new = horizontal_filter;
20479+ resize->horizontal_filter_support_new = horizontal_support;
20480+ resize->horizontal_filter_radius = horizontal_radius;
20481+ resize->horizontal_filter_kernel = 0; // (the old form's, if it was set before)
20482+ resize->horizontal_filter_support = 0;
20483+ resize->vertical_filter_kernel_new = vertical_filter;
20484+ resize->vertical_filter_support_new = vertical_support;
20485+ resize->vertical_filter_radius = vertical_radius;
20486+ resize->vertical_filter_kernel = 0;
20487+ resize->vertical_filter_support = 0;
20488+ resize->needs_rebuild = 1;
20489+ return 1;
20490+}
20491+
20492+// the old form: the float kernels and supports are kept apart and the sampler calls them through a conversion
20493+STBIRDEF int stbir_set_filter_callbacks( STBIR_RESIZE * resize, stbir__kernel_callback * horizontal_filter, stbir__support_callback * horizontal_support, stbir__kernel_callback * vertical_filter, stbir__support_callback * vertical_support )
20494+{
20495+ resize->horizontal_filter_kernel = horizontal_filter;
20496+ resize->horizontal_filter_support = horizontal_support;
20497+ resize->horizontal_filter_kernel_new = 0;
20498+ resize->horizontal_filter_support_new = 0;
20499+ resize->horizontal_filter_radius = 0;
20500+ resize->vertical_filter_kernel = vertical_filter;
20501+ resize->vertical_filter_support = vertical_support;
20502+ resize->vertical_filter_kernel_new = 0;
20503+ resize->vertical_filter_support_new = 0;
20504+ resize->vertical_filter_radius = 0;
20505+ resize->needs_rebuild = 1;
20506+ return 1;
20507+}
20508+
20509+STBIRDEF int stbir_set_pixel_layouts( STBIR_RESIZE * resize, stbir_pixel_layout input_pixel_layout, stbir_pixel_layout output_pixel_layout ) // sets new pixel layouts
20510+{
20511+ resize->input_pixel_layout_public = input_pixel_layout;
20512+ resize->output_pixel_layout_public = output_pixel_layout;
20513+ resize->needs_rebuild = 1;
20514+ return 1;
20515+}
20516+
20517+
20518+STBIRDEF int stbir_set_non_pm_alpha_speed_over_quality( STBIR_RESIZE * resize, int non_pma_alpha_speed_over_quality ) // sets alpha speed
20519+{
20520+ resize->fast_alpha = non_pma_alpha_speed_over_quality;
20521+ resize->needs_rebuild = 1;
20522+ return 1;
20523+}
20524+
20525+STBIRDEF int stbir_set_input_subrect( STBIR_RESIZE * resize, stbir_real s0, stbir_real t0, stbir_real s1, stbir_real t1 ) // sets input region (full region by default)
20526+{
20527+ resize->input_s0 = s0;
20528+ resize->input_t0 = t0;
20529+ resize->input_s1 = s1;
20530+ resize->input_t1 = t1;
20531+ resize->input_s_d = 0; // not in pixels: the fractions are worked out from the reals at build time
20532+ resize->input_t_d = 0;
20533+ resize->needs_rebuild = 1;
20534+
20535+ // are we inbounds? (a zero or negative area, or a region wholly past the image, is empty)
20536+ if ( ( s1 <= (stbir_real)0.0 ) || ( s1 <= s0 ) ||
20537+ ( t1 <= (stbir_real)0.0 ) || ( t1 <= t0 ) ||
20538+ ( s0 >= (stbir_real)1.0 ) ||
20539+ ( t0 >= (stbir_real)1.0 ) )
20540+ return 0;
20541+
20542+ return 1;
20543+}
20544+
20545+STBIRDEF int stbir_set_output_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ) // sets the output sub rect, in output pixels (the whole output by default)
20546+{
20547+ resize->output_subx = subx;
20548+ resize->output_suby = suby;
20549+ resize->output_subw = subw;
20550+ resize->output_subh = subh;
20551+ resize->needs_rebuild = 1;
20552+
20553+ // is there anything in it? (a negative size is an empty rect - the resize then has nothing to do)
20554+ if ( ( subx >= resize->output_w ) || ( ( subx + subw ) <= 0 ) || ( suby >= resize->output_h ) || ( ( suby + subh ) <= 0 ) || ( subw <= 0 ) || ( subh <= 0 ) )
20555+ return 0;
20556+
20557+ return 1;
20558+}
20559+
20560+STBIRDEF int stbir_set_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ) // sets both regions (full regions by default)
20561+{
20562+ stbir_real s0, t0, s1, t1;
20563+
20564+ s0 = ( (stbir_real)subx ) / ( (stbir_real)resize->output_w );
20565+ t0 = ( (stbir_real)suby ) / ( (stbir_real)resize->output_h );
20566+ s1 = ( (stbir_real)(subx+subw) ) / ( (stbir_real)resize->output_w );
20567+ t1 = ( (stbir_real)(suby+subh) ) / ( (stbir_real)resize->output_h );
20568+
20569+ resize->input_s0 = s0;
20570+ resize->input_t0 = t0;
20571+ resize->input_s1 = s1;
20572+ resize->input_t1 = t1;
20573+ resize->input_s_n0 = subx;
20574+ resize->input_s_n1 = subx + subw;
20575+ resize->input_s_d = resize->output_w;
20576+ resize->input_t_n0 = suby;
20577+ resize->input_t_n1 = suby + subh;
20578+ resize->input_t_d = resize->output_h;
20579+ resize->output_subx = subx;
20580+ resize->output_suby = suby;
20581+ resize->output_subw = subw;
20582+ resize->output_subh = subh;
20583+ resize->needs_rebuild = 1;
20584+
20585+ // is there anything in it? (a negative size is an empty rect - the resize then has nothing to do)
20586+ if ( ( subx >= resize->output_w ) || ( ( subx + subw ) <= 0 ) || ( suby >= resize->output_h ) || ( ( suby + subh ) <= 0 ) || ( subw <= 0 ) || ( subh <= 0 ) )
20587+ return 0;
20588+
20589+ return 1;
20590+}
20591+
20592+static int stbir__perform_build( STBIR_RESIZE * resize, int splits )
20593+{
20594+ stbir__contributors conservative = { 0, 0 };
20595+ stbir__sampler horizontal, vertical;
20596+ int new_output_subx, new_output_suby;
20597+ stbir__info * out_info;
20598+ #ifdef STBIR_PROFILE
20599+ stbir__info profile_infod; // used to contain building profile info before everything is allocated
20600+ stbir__info * profile_info = &profile_infod;
20601+ #endif
20602+
20603+ // have we already built the samplers?
20604+ if ( resize->samplers )
20605+ return 0;
20606+
20607+ #define STBIR_RETURN_ERROR_AND_ASSERT( exp ) STBIR_ASSERT( !(exp) ); if (exp) return 0;
20608+ STBIR_RETURN_ERROR_AND_ASSERT( (unsigned)resize->horizontal_filter >= STBIR_FILTER_OTHER)
20609+ STBIR_RETURN_ERROR_AND_ASSERT( (unsigned)resize->vertical_filter >= STBIR_FILTER_OTHER)
20610+ #undef STBIR_RETURN_ERROR_AND_ASSERT
20611+
20612+ if ( splits <= 0 )
20613+ return 0;
20614+
20615+ STBIR_PROFILE_BUILD_FIRST_START( build );
20616+
20617+ new_output_subx = resize->output_subx;
20618+ new_output_suby = resize->output_suby;
20619+
20620+ // do horizontal clip and scale calcs (an empty region - zero or negative sizes, nothing in bounds - fails the build)
20621+ if ( !stbir__calculate_region_transform( &horizontal.scale_info, resize->output_w, &new_output_subx, resize->output_subw, resize->input_w, resize->input_s0, resize->input_s1, resize->input_s_n0, resize->input_s_n1, resize->input_s_d ) )
20622+ return 0;
20623+
20624+ // do vertical clip and scale calcs
20625+ if ( !stbir__calculate_region_transform( &vertical.scale_info, resize->output_h, &new_output_suby, resize->output_subh, resize->input_h, resize->input_t0, resize->input_t1, resize->input_t_n0, resize->input_t_n1, resize->input_t_d ) )
20626+ return 0;
20627+
20628+ // if nothing to do, just return
20629+ if ( ( horizontal.scale_info.output_sub_size == 0 ) || ( vertical.scale_info.output_sub_size == 0 ) )
20630+ return 0;
20631+
20632+ stbir__set_sampler(&horizontal, resize->horizontal_filter, resize->horizontal_filter_kernel_new, resize->horizontal_filter_support_new, resize->horizontal_filter_radius, resize->horizontal_filter_kernel, resize->horizontal_filter_support, resize->horizontal_edge, &horizontal.scale_info, 1, resize->user_data );
20633+ stbir__get_conservative_extents( &horizontal, &conservative, resize->user_data );
20634+ stbir__set_sampler(&vertical, resize->vertical_filter, resize->vertical_filter_kernel_new, resize->vertical_filter_support_new, resize->vertical_filter_radius, resize->vertical_filter_kernel, resize->vertical_filter_support, resize->vertical_edge, &vertical.scale_info, 0, resize->user_data );
20635+
20636+ // the split count can come down in the build (stbir__choose_split_grid): the caller gets the count it can use
20637+
20638+ STBIR_PROFILE_BUILD_START( alloc );
20639+ out_info = stbir__alloc_internal_mem_and_build_samplers( &horizontal, &vertical, &conservative, resize->input_pixel_layout_public, resize->output_pixel_layout_public, splits, new_output_subx, new_output_suby, resize->fast_alpha, stbir__pixel_channels[ stbir__pixel_layout_convert_public_to_internal[ resize->input_pixel_layout_public ] ] * stbir__type_size[ resize->input_data_type ], resize->user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
20640+ STBIR_PROFILE_BUILD_END( alloc );
20641+ STBIR_PROFILE_BUILD_END( build );
20642+
20643+ if ( out_info )
20644+ {
20645+ splits = out_info->splits;
20646+ resize->splits = splits;
20647+ resize->samplers = out_info;
20648+ resize->needs_rebuild = 0;
20649+ #ifdef STBIR_PROFILE
20650+ STBIR_MEMCPY( &out_info->profile, &profile_infod.profile, sizeof( out_info->profile ) );
20651+ #endif
20652+
20653+ // update anything that can be changed without recalcing samplers
20654+ stbir__update_info_from_resize( out_info, resize );
20655+
20656+ return splits;
20657+ }
20658+
20659+ return 0;
20660+}
20661+
20662+STBIRDEF void stbir_free_samplers( STBIR_RESIZE * resize )
20663+{
20664+ if ( resize->samplers )
20665+ {
20666+ stbir__free_internal_mem( resize->samplers );
20667+ resize->samplers = 0;
20668+ resize->called_alloc = 0;
20669+ }
20670+}
20671+
20672+STBIRDEF int stbir_build_samplers_with_splits( STBIR_RESIZE * resize, int splits )
20673+{
20674+ if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
20675+ {
20676+ if ( resize->samplers )
20677+ stbir_free_samplers( resize );
20678+
20679+ resize->called_alloc = 1;
20680+ return stbir__perform_build( resize, splits );
20681+ }
20682+
20683+ STBIR_PROFILE_BUILD_CLEAR( resize->samplers );
20684+
20685+ return resize->splits; // already built: the count that build settled on (a new request isn't applied without a rebuild)
20686+}
20687+
20688+STBIRDEF int stbir_build_samplers( STBIR_RESIZE * resize )
20689+{
20690+ return stbir_build_samplers_with_splits( resize, 1 );
20691+}
20692+
20693+STBIRDEF int stbir_resize_extended( STBIR_RESIZE * resize )
20694+{
20695+ int result;
20696+
20697+ if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
20698+ {
20699+ int alloc_state = resize->called_alloc; // remember allocated state
20700+
20701+ if ( resize->samplers )
20702+ {
20703+ stbir__free_internal_mem( resize->samplers );
20704+ resize->samplers = 0;
20705+ }
20706+
20707+ if ( !stbir_build_samplers( resize ) )
20708+ return 0; // (an empty resize fails the build too, so it comes out of here as a failure)
20709+
20710+ resize->called_alloc = alloc_state;
20711+ }
20712+ else
20713+ {
20714+ // didn't build anything - clear it
20715+ STBIR_PROFILE_BUILD_CLEAR( resize->samplers );
20716+ }
20717+
20718+ // do resize
20719+ result = stbir__perform_resize( resize->samplers, 0, resize->splits );
20720+
20721+ // if we alloced, then free
20722+ if ( !resize->called_alloc )
20723+ {
20724+ stbir_free_samplers( resize );
20725+ resize->samplers = 0;
20726+ }
20727+
20728+ return result;
20729+}
20730+
20731+STBIRDEF int stbir_resize_extended_split( STBIR_RESIZE * resize, int split_start, int split_count )
20732+{
20733+ STBIR_ASSERT( resize->samplers );
20734+
20735+ // you **must** build samplers first when using split resize (and not change the resize after)
20736+ if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
20737+ return 0;
20738+
20739+ // if we're just doing the whole thing, call full
20740+ if ( ( split_start == -1 ) || ( ( split_start == 0 ) && ( split_count == resize->splits ) ) )
20741+ return stbir_resize_extended( resize );
20742+
20743+ if ( ( split_start >= resize->splits ) || ( split_start < 0 ) || ( ( split_start + split_count ) > resize->splits ) || ( split_count <= 0 ) )
20744+ return 0;
20745+
20746+ // do resize
20747+ return stbir__perform_resize( resize->samplers, split_start, split_count );
20748+}
20749+
20750+
20751+static void * stbir_quick_resize_helper( const void *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
20752+ void *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
20753+ stbir_pixel_layout pixel_layout, stbir_datatype data_type, stbir_edge edge, stbir_filter filter )
20754+{
20755+ STBIR_RESIZE resize;
20756+ stbir_int64 scanline_output_in_bytes;
20757+ stbir_int64 positive_output_stride_in_bytes;
20758+ void * start_ptr;
20759+ void * free_ptr;
20760+
20761+ scanline_output_in_bytes = (stbir_int64)output_w * stbir__type_size[ data_type ] * stbir__pixel_channels[ stbir__pixel_layout_convert_public_to_internal[ pixel_layout ] ];
20762+ if ( scanline_output_in_bytes <= 0 )
20763+ return 0;
20764+
20765+ // if zero stride, use scanline output
20766+ if ( output_stride_in_bytes == 0 )
20767+ output_stride_in_bytes = scanline_output_in_bytes;
20768+
20769+ // abs value for inverted images (negative pitches)
20770+ positive_output_stride_in_bytes = output_stride_in_bytes;
20771+ if ( positive_output_stride_in_bytes < 0 )
20772+ positive_output_stride_in_bytes = -positive_output_stride_in_bytes;
20773+
20774+ // is the requested stride smaller than the scanline output? if so, just fail
20775+ if ( positive_output_stride_in_bytes < scanline_output_in_bytes )
20776+ return 0;
20777+
20778+ start_ptr = output_pixels;
20779+ free_ptr = 0; // no free pointer, since they passed buffer to use
20780+
20781+ // did they pass a zero for the dest? if so, allocate the buffer
20782+ if ( output_pixels == 0 )
20783+ {
20784+ size_t size;
20785+ char * ptr;
20786+
20787+ size = (size_t)positive_output_stride_in_bytes * (size_t)output_h;
20788+ if ( size == 0 )
20789+ return 0;
20790+
20791+ ptr = (char*) STBIR_MALLOC( size, 0 );
20792+ if ( ptr == 0 )
20793+ return 0;
20794+
20795+ free_ptr = ptr;
20796+
20797+ // point at the last scanline, if they requested a flipped image
20798+ if ( output_stride_in_bytes < 0 )
20799+ start_ptr = ptr + ( (size_t)positive_output_stride_in_bytes * (size_t)( output_h - 1 ) );
20800+ else
20801+ start_ptr = ptr;
20802+ }
20803+
20804+ // ok, now do the resize
20805+ stbir_resize_init( &resize,
20806+ input_pixels, input_w, input_h, input_stride_in_bytes,
20807+ start_ptr, output_w, output_h, output_stride_in_bytes,
20808+ pixel_layout, data_type );
20809+
20810+ resize.horizontal_edge = edge;
20811+ resize.vertical_edge = edge;
20812+ resize.horizontal_filter = filter;
20813+ resize.vertical_filter = filter;
20814+
20815+ if ( !stbir_resize_extended( &resize ) )
20816+ {
20817+ if ( free_ptr )
20818+ STBIR_FREE( free_ptr, 0 );
20819+ return 0;
20820+ }
20821+
20822+ return (free_ptr) ? free_ptr : start_ptr;
20823+}
20824+
20825+
20826+STBIRDEF unsigned char * stbir_resize_uint8_linear( const unsigned char *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
20827+ unsigned char *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
20828+ stbir_pixel_layout pixel_layout )
20829+{
20830+ return (unsigned char *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
20831+ output_pixels, output_w, output_h, output_stride_in_bytes,
20832+ pixel_layout, STBIR_TYPE_UINT8, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
20833+}
20834+
20835+STBIRDEF unsigned char * stbir_resize_uint8_srgb( const unsigned char *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
20836+ unsigned char *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
20837+ stbir_pixel_layout pixel_layout )
20838+{
20839+ return (unsigned char *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
20840+ output_pixels, output_w, output_h, output_stride_in_bytes,
20841+ pixel_layout, STBIR_TYPE_UINT8_SRGB, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
20842+}
20843+
20844+
20845+STBIRDEF float * stbir_resize_float_linear( const float *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
20846+ float *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
20847+ stbir_pixel_layout pixel_layout )
20848+{
20849+ return (float *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
20850+ output_pixels, output_w, output_h, output_stride_in_bytes,
20851+ pixel_layout, STBIR_TYPE_FLOAT, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
20852+}
20853+
20854+
20855+STBIRDEF void * stbir_resize( const void *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
20856+ void *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
20857+ stbir_pixel_layout pixel_layout, stbir_datatype data_type,
20858+ stbir_edge edge, stbir_filter filter )
20859+{
20860+ return (void *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
20861+ output_pixels, output_w, output_h, output_stride_in_bytes,
20862+ pixel_layout, data_type, edge, filter );
20863+}
20864+
20865+#ifdef STBIR_PROFILE
20866+
20867+STBIRDEF void stbir_resize_build_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize )
20868+{
20869+ static char const * bdescriptions[6] = { "Building", "Allocating", "Horizontal sampler", "Vertical sampler", "Coefficient cleanup", "Scatter coefficients" } ;
20870+ stbir__info* samp = resize->samplers;
20871+ int i;
20872+
20873+ for( i = 0 ; i < (int)STBIR__ARRAY_SIZE( bdescriptions ) ; i++)
20874+ info->clocks[i] = samp->profile.array[i+1];
20875+
20876+ info->total_clocks = samp->profile.named.total;
20877+ info->descriptions = bdescriptions;
20878+ info->count = STBIR__ARRAY_SIZE( bdescriptions );
20879+}
20880+
20881+STBIRDEF void stbir_resize_split_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize, int split_start, int split_count )
20882+{
20883+ static char const * descriptions[7] = { "Looping", "Vertical sampling", "Horizontal sampling", "Scanline input", "Scanline output", "Alpha weighting", "Alpha unweighting" };
20884+ stbir__per_split_info * split_info;
20885+ int s, i;
20886+
20887+ if ( split_start == -1 )
20888+ {
20889+ split_start = 0;
20890+ split_count = resize->samplers->splits;
20891+ }
20892+
20893+ if ( ( split_start >= resize->splits ) || ( split_start < 0 ) || ( ( split_start + split_count ) > resize->splits ) || ( split_count <= 0 ) )
20894+ {
20895+ info->total_clocks = 0;
20896+ info->descriptions = 0;
20897+ info->count = 0;
20898+ return;
20899+ }
20900+
20901+ split_info = resize->samplers->split_info + split_start;
20902+
20903+ // sum up the profile from all the splits
20904+ for( i = 0 ; i < (int)STBIR__ARRAY_SIZE( descriptions ) ; i++ )
20905+ {
20906+ stbir_uint64 sum = 0;
20907+ for( s = 0 ; s < split_count ; s++ )
20908+ sum += split_info[s].profile.array[i+1];
20909+ info->clocks[i] = sum;
20910+ }
20911+
20912+ info->total_clocks = 0;
20913+ for( s = 0 ; s < split_count ; s++ )
20914+ info->total_clocks += split_info[s].profile.named.total;
20915+ info->descriptions = descriptions;
20916+ info->count = STBIR__ARRAY_SIZE( descriptions );
20917+}
20918+
20919+STBIRDEF void stbir_resize_extended_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize )
20920+{
20921+ stbir_resize_split_profile_info( info, resize, -1, 0 );
20922+}
20923+
20924+#endif // STBIR_PROFILE
20925+
20926+#undef STBIR_BGR
20927+#undef STBIR_1CHANNEL
20928+#undef STBIR_2CHANNEL
20929+#undef STBIR_RGB
20930+#undef STBIR_RGBA
20931+#undef STBIR_4CHANNEL
20932+#undef STBIR_BGRA
20933+#undef STBIR_ARGB
20934+#undef STBIR_ABGR
20935+#undef STBIR_RA
20936+#undef STBIR_AR
20937+#undef STBIR_RGBA_PM
20938+#undef STBIR_BGRA_PM
20939+#undef STBIR_ARGB_PM
20940+#undef STBIR_ABGR_PM
20941+#undef STBIR_RA_PM
20942+#undef STBIR_AR_PM
20943+
20944+#endif // STB_IMAGE_RESIZE_IMPLEMENTATION
20945+
20946+#else // STB_IMAGE_RESIZE_HORIZONTALS&STB_IMAGE_RESIZE_DO_VERTICALS
20947+
20948+// we reinclude the header file to define all the horizontal functions
20949+// specializing each function for the number of coeffs is 20-40% faster *OVERALL*
20950+
20951+// by including the header file again this way, we can still debug the functions
20952+
20953+#define STBIR_strs_join2( start, mid, end ) start##mid##end
20954+#define STBIR_strs_join1( start, mid, end ) STBIR_strs_join2( start, mid, end )
20955+
20956+#define STBIR_strs_join24( start, mid1, mid2, end ) start##mid1##mid2##end
20957+#define STBIR_strs_join14( start, mid1, mid2, end ) STBIR_strs_join24( start, mid1, mid2, end )
20958+
20959+#ifdef STB_IMAGE_RESIZE_DO_CODERS
20960+
20961+#ifdef stbir__decode_suffix
20962+#define STBIR__CODER_NAME( name ) STBIR_strs_join1( name, _, stbir__decode_suffix )
20963+#else
20964+#define STBIR__CODER_NAME( name ) name
20965+#endif
20966+
20967+#ifdef stbir__decode_swizzle
20968+#define stbir__decode_simdf8_flip(reg) STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( stbir__simdf8_0123to,stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3),stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3)(reg, reg)
20969+#define stbir__decode_simdf4_flip(reg) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf_0123to,stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3)(reg, reg)
20970+#define stbir__encode_simdf8_unflip(reg) STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( stbir__simdf8_0123to,stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3),stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3)(reg, reg)
20971+#define stbir__encode_simdf4_unflip(reg) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf_0123to,stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3)(reg, reg)
20972+#define stbir__encode_simdf16_unflip(reg) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf16_0123to,stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3)(reg, reg) // (one lane's order, the shuffle repeats it)
20973+// u8 and u16 inputs are swizzled as bytes before the expansion when there is a byte shuffle
20974+#ifdef STBIR_SIMD_BYTE_SHUFFLE
20975+#define STBIR__SWZ8( p ) ( ( (p)*4 + stbir__decode_order0 ) | ( ( (p)*4 + stbir__decode_order1 ) << 8 ) | ( ( (p)*4 + stbir__decode_order2 ) << 16 ) | ( ( (p)*4 + stbir__decode_order3 ) << 24 ) )
20976+#define STBIR__SWZ16( p, a, b ) ( ( 2*((p)*4 + (a)) ) | ( ( 2*((p)*4 + (a)) + 1 ) << 8 ) | ( ( 2*((p)*4 + (b)) ) << 16 ) | ( ( 2*((p)*4 + (b)) + 1 ) << 24 ) )
20977+static STBIR__SIMDI_CONST4( STBIR__CODER_NAME( stbir__decode_bytes_swizzle ), STBIR__SWZ8(0), STBIR__SWZ8(1), STBIR__SWZ8(2), STBIR__SWZ8(3) );
20978+static STBIR__SIMDI_CONST4( STBIR__CODER_NAME( stbir__decode_words_swizzle ), STBIR__SWZ16(0,stbir__decode_order0,stbir__decode_order1), STBIR__SWZ16(0,stbir__decode_order2,stbir__decode_order3), STBIR__SWZ16(1,stbir__decode_order0,stbir__decode_order1), STBIR__SWZ16(1,stbir__decode_order2,stbir__decode_order3) );
20979+#define stbir__decode_simdi_flip_u8( ireg ) stbir__simdi_shuffle_bytes( ireg, ireg, STBIR__CONSTI( STBIR__CODER_NAME( stbir__decode_bytes_swizzle ) ) )
20980+#define stbir__decode_simdi_flip_u16( ireg ) stbir__simdi_shuffle_bytes( ireg, ireg, STBIR__CONSTI( STBIR__CODER_NAME( stbir__decode_words_swizzle ) ) )
20981+#define stbir__decode_simdf8_flip_i( reg )
20982+#define stbir__decode_simdf4_flip_i( reg )
20983+#define stbir__decode_simdf16_flip_i( reg )
20984+#ifdef STBIR_SIMD16
20985+#define stbir__decode_simdi8_flip_u16( ireg ) stbir__simdi8_shuffle_bytes( ireg, ireg, STBIR__CONSTI( STBIR__CODER_NAME( stbir__decode_words_swizzle ) ) ) // (the 16-wide u16 decoder loads 32 bytes: the same table in both lanes)
20986+#endif
20987+#else
20988+#define stbir__decode_simdi_flip_u8( ireg )
20989+#define stbir__decode_simdi_flip_u16( ireg )
20990+#define stbir__decode_simdi8_flip_u16( ireg )
20991+#define stbir__decode_simdf8_flip_i( reg ) stbir__decode_simdf8_flip( reg )
20992+#define stbir__decode_simdf4_flip_i( reg ) stbir__decode_simdf4_flip( reg )
20993+#define stbir__decode_simdf16_flip_i( reg ) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf16_0123to,stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3)(reg, reg)
20994+#endif
20995+#else
20996+#define stbir__decode_order0 0
20997+#define stbir__decode_order1 1
20998+#define stbir__decode_order2 2
20999+#define stbir__decode_order3 3
21000+#define stbir__encode_order0 0
21001+#define stbir__encode_order1 1
21002+#define stbir__encode_order2 2
21003+#define stbir__encode_order3 3
21004+#define stbir__decode_simdf8_flip(reg)
21005+#define stbir__decode_simdf4_flip(reg)
21006+#define stbir__decode_simdi_flip_u8( ireg )
21007+#define stbir__decode_simdi_flip_u16( ireg )
21008+#define stbir__decode_simdi8_flip_u16( ireg )
21009+#define stbir__decode_simdf8_flip_i( reg )
21010+#define stbir__decode_simdf4_flip_i( reg )
21011+#define stbir__decode_simdf16_flip_i( reg )
21012+#define stbir__encode_simdf8_unflip(reg)
21013+#define stbir__encode_simdf4_unflip(reg)
21014+#define stbir__encode_simdf16_unflip(reg)
21015+#endif
21016+
21017+#if defined(STBIR_SIMD16)
21018+#define stbir__encode_simdfX_unflip stbir__encode_simdf16_unflip
21019+#elif defined(STBIR_SIMD8)
21020+#define stbir__encode_simdfX_unflip stbir__encode_simdf8_unflip
21021+#else
21022+#define stbir__encode_simdfX_unflip stbir__encode_simdf4_unflip
21023+#endif
21024+
21025+static float * STBIR__CODER_NAME( stbir__decode_uint8_linear_scaled )( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
21026+{
21027+ float STBIR_STREAMOUT_PTR( * ) decode = decodep;
21028+ float * decode_end = (float*) decode + width_times_channels;
21029+ unsigned char const * input = (unsigned char const*)inputp;
21030+
21031+ #ifdef STBIR_SIMD
21032+ unsigned char const * end_input_m16 = input + width_times_channels - 16;
21033+ if ( width_times_channels >= 16 )
21034+ {
21035+ decode_end -= 16;
21036+ STBIR_NO_UNROLL_LOOP_START_INF_FOR
21037+ for(;;)
21038+ {
21039+ #if defined(STBIR_SIMD16)
21040+ stbir__simdi i; stbir__simdi16 o;
21041+ stbir__simdf16 of;
21042+ STBIR_NO_UNROLL(decode);
21043+ stbir__simdi_load( i, input );
21044+ stbir__decode_simdi_flip_u8( i );
21045+ stbir__simdi16_expand_u8_to_u32( o, i );
21046+ stbir__simdi16_convert_i32_to_float( of, o );
21047+ stbir__simdf16_mult( of, of, STBIR_max_uint8_as_float_inverted16 );
21048+ stbir__decode_simdf16_flip_i( of );
21049+ stbir__simdf16_store( decode, of );
21050+ #elif defined(STBIR_SIMD8)
21051+ stbir__simdi i; stbir__simdi8 o0,o1;
21052+ stbir__simdf8 of0, of1;
21053+ STBIR_NO_UNROLL(decode);
21054+ stbir__simdi_load( i, input );
21055+ stbir__decode_simdi_flip_u8( i );
21056+ stbir__simdi8_expand_u8_to_u32( o0, o1, i );
21057+ stbir__simdi8_convert_i32_to_float( of0, o0 );
21058+ stbir__simdi8_convert_i32_to_float( of1, o1 );
21059+ stbir__simdf8_mult( of0, of0, STBIR_max_uint8_as_float_inverted8);
21060+ stbir__simdf8_mult( of1, of1, STBIR_max_uint8_as_float_inverted8);
21061+ stbir__decode_simdf8_flip_i( of0 );
21062+ stbir__decode_simdf8_flip_i( of1 );
21063+ stbir__simdf8_store( decode + 0, of0 );
21064+ stbir__simdf8_store( decode + 8, of1 );
21065+ #else
21066+ stbir__simdi i, o0, o1, o2, o3;
21067+ stbir__simdf of0, of1, of2, of3;
21068+ STBIR_NO_UNROLL(decode);
21069+ stbir__simdi_load( i, input );
21070+ stbir__decode_simdi_flip_u8( i );
21071+ stbir__simdi_expand_u8_to_u32( o0,o1,o2,o3,i);
21072+ stbir__simdi_convert_i32_to_float( of0, o0 );
21073+ stbir__simdi_convert_i32_to_float( of1, o1 );
21074+ stbir__simdi_convert_i32_to_float( of2, o2 );
21075+ stbir__simdi_convert_i32_to_float( of3, o3 );
21076+ stbir__simdf_mult( of0, of0, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
21077+ stbir__simdf_mult( of1, of1, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
21078+ stbir__simdf_mult( of2, of2, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
21079+ stbir__simdf_mult( of3, of3, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
21080+ stbir__decode_simdf4_flip_i( of0 );
21081+ stbir__decode_simdf4_flip_i( of1 );
21082+ stbir__decode_simdf4_flip_i( of2 );
21083+ stbir__decode_simdf4_flip_i( of3 );
21084+ stbir__simdf_store( decode + 0, of0 );
21085+ stbir__simdf_store( decode + 4, of1 );
21086+ stbir__simdf_store( decode + 8, of2 );
21087+ stbir__simdf_store( decode + 12, of3 );
21088+ #endif
21089+ decode += 16;
21090+ input += 16;
21091+ if ( decode <= decode_end )
21092+ continue;
21093+ if ( decode == ( decode_end + 16 ) )
21094+ break;
21095+ decode = decode_end; // backup and do last couple
21096+ input = end_input_m16;
21097+ }
21098+ return decode_end + 16;
21099+ }
21100+ #endif
21101+
21102+ // try to do blocks of 4 when you can
21103+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
21104+ decode += 4;
21105+ STBIR_SIMD_NO_UNROLL_LOOP_START
21106+ while( decode <= decode_end )
21107+ {
21108+ STBIR_SIMD_NO_UNROLL(decode);
21109+ decode[0-4] = ((float)(input[stbir__decode_order0])) * stbir__max_uint8_as_float_inverted;
21110+ decode[1-4] = ((float)(input[stbir__decode_order1])) * stbir__max_uint8_as_float_inverted;
21111+ decode[2-4] = ((float)(input[stbir__decode_order2])) * stbir__max_uint8_as_float_inverted;
21112+ decode[3-4] = ((float)(input[stbir__decode_order3])) * stbir__max_uint8_as_float_inverted;
21113+ decode += 4;
21114+ input += 4;
21115+ }
21116+ decode -= 4;
21117+ #endif
21118+
21119+ // do the remnants
21120+ #if stbir__coder_min_num < 4
21121+ STBIR_NO_UNROLL_LOOP_START
21122+ while( decode < decode_end )
21123+ {
21124+ STBIR_NO_UNROLL(decode);
21125+ decode[0] = ((float)(input[stbir__decode_order0])) * stbir__max_uint8_as_float_inverted;
21126+ #if stbir__coder_min_num >= 2
21127+ decode[1] = ((float)(input[stbir__decode_order1])) * stbir__max_uint8_as_float_inverted;
21128+ #endif
21129+ #if stbir__coder_min_num >= 3
21130+ decode[2] = ((float)(input[stbir__decode_order2])) * stbir__max_uint8_as_float_inverted;
21131+ #endif
21132+ decode += stbir__coder_min_num;
21133+ input += stbir__coder_min_num;
21134+ }
21135+ #endif
21136+
21137+ return decode_end;
21138+}
21139+
21140+static void STBIR__CODER_NAME( stbir__encode_uint8_linear_scaled )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
21141+{
21142+ unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char *) outputp;
21143+ unsigned char * end_output = ( (unsigned char *) output ) + width_times_channels;
21144+
21145+ #ifdef STBIR_SIMD
21146+ // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
21147+ // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
21148+ STBIR_SIMD_NO_UNROLL_LOOP_START
21149+ while ( ( end_output - output ) >= stbir__simdfX_float_count*2 )
21150+ {
21151+ stbir__simdfX e0, e1;
21152+ STBIR_SIMD_NO_UNROLL(encode);
21153+ stbir__simdfX_madd_mem( e0, STBIR_simd_point5X, STBIR_max_uint8_as_floatX, encode );
21154+ stbir__simdfX_madd_mem( e1, STBIR_simd_point5X, STBIR_max_uint8_as_floatX, encode+stbir__simdfX_float_count );
21155+ stbir__encode_simdfX_unflip( e0 );
21156+ stbir__encode_simdfX_unflip( e1 );
21157+ #if defined(STBIR_SIMD16)
21158+ {
21159+ stbir__simdi8 i;
21160+ stbir__simdf16_pack_to_32bytes( i, e0, e1 );
21161+ stbir__simdi8_store( output, i );
21162+ }
21163+ #elif defined(STBIR_SIMD8)
21164+ {
21165+ stbir__simdi i;
21166+ stbir__simdf8_pack_to_16bytes( i, e0, e1 );
21167+ stbir__simdi_store( output, i );
21168+ }
21169+ #else
21170+ {
21171+ stbir__simdi i;
21172+ stbir__simdf_pack_to_8bytes( i, e0, e1 );
21173+ stbir__simdi_store2( output, i );
21174+ }
21175+ #endif
21176+ encode += stbir__simdfX_float_count*2;
21177+ output += stbir__simdfX_float_count*2;
21178+ }
21179+
21180+ // try to do blocks of 4 when you can
21181+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
21182+ output += 4;
21183+ STBIR_NO_UNROLL_LOOP_START
21184+ while( output <= end_output )
21185+ {
21186+ stbir__simdf e0;
21187+ stbir__simdi i0;
21188+ STBIR_NO_UNROLL(encode);
21189+ stbir__simdf_load( e0, encode );
21190+ stbir__simdf_madd( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), e0 );
21191+ stbir__encode_simdf4_unflip( e0 );
21192+ stbir__simdf_pack_to_8bytes( i0, e0, e0 ); // only use first 4
21193+ *(int*)(output-4) = stbir__simdi_to_int( i0 );
21194+ output += 4;
21195+ encode += 4;
21196+ }
21197+ output -= 4;
21198+ #endif
21199+
21200+ // do the remnants
21201+ #if stbir__coder_min_num < 4
21202+ STBIR_NO_UNROLL_LOOP_START
21203+ while( output < end_output )
21204+ {
21205+ // read the whole pixel before writing any of it - the encode can be in place, and a swizzle reads a channel after
21206+ // the one it lands on has been overwritten
21207+ stbir__simdf e0;
21208+ #if stbir__coder_min_num >= 2
21209+ stbir__simdf e1;
21210+ #endif
21211+ #if stbir__coder_min_num >= 3
21212+ stbir__simdf e2;
21213+ #endif
21214+ STBIR_NO_UNROLL(encode);
21215+ stbir__simdf_madd1_mem( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order0 );
21216+ #if stbir__coder_min_num >= 2
21217+ stbir__simdf_madd1_mem( e1, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order1 );
21218+ #endif
21219+ #if stbir__coder_min_num >= 3
21220+ stbir__simdf_madd1_mem( e2, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order2 );
21221+ #endif
21222+ output[0] = stbir__simdf_convert_float_to_uint8( e0 );
21223+ #if stbir__coder_min_num >= 2
21224+ output[1] = stbir__simdf_convert_float_to_uint8( e1 );
21225+ #endif
21226+ #if stbir__coder_min_num >= 3
21227+ output[2] = stbir__simdf_convert_float_to_uint8( e2 );
21228+ #endif
21229+ output += stbir__coder_min_num;
21230+ encode += stbir__coder_min_num;
21231+ }
21232+ #endif
21233+
21234+ #else
21235+
21236+ // try to do blocks of 4 when you can
21237+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
21238+ output += 4;
21239+ while( output <= end_output )
21240+ {
21241+ float f0 = encode[stbir__encode_order0] * stbir__max_uint8_as_float + 0.5f;
21242+ float f1 = encode[stbir__encode_order1] * stbir__max_uint8_as_float + 0.5f;
21243+ float f2 = encode[stbir__encode_order2] * stbir__max_uint8_as_float + 0.5f;
21244+ float f3 = encode[stbir__encode_order3] * stbir__max_uint8_as_float + 0.5f;
21245+ STBIR_CLAMP(f0, 0, 255);
21246+ STBIR_CLAMP(f1, 0, 255);
21247+ STBIR_CLAMP(f2, 0, 255);
21248+ STBIR_CLAMP(f3, 0, 255);
21249+ output[0-4] = (unsigned char)f0;
21250+ output[1-4] = (unsigned char)f1;
21251+ output[2-4] = (unsigned char)f2;
21252+ output[3-4] = (unsigned char)f3;
21253+ output += 4;
21254+ encode += 4;
21255+ }
21256+ output -= 4;
21257+ #endif
21258+
21259+ // do the remnants
21260+ #if stbir__coder_min_num < 4
21261+ STBIR_NO_UNROLL_LOOP_START
21262+ while( output < end_output )
21263+ {
21264+ float f0 = encode[stbir__encode_order0] * stbir__max_uint8_as_float + 0.5f;
21265+ #if stbir__coder_min_num >= 2
21266+ float f1 = encode[stbir__encode_order1] * stbir__max_uint8_as_float + 0.5f;
21267+ #endif
21268+ #if stbir__coder_min_num >= 3
21269+ float f2 = encode[stbir__encode_order2] * stbir__max_uint8_as_float + 0.5f;
21270+ #endif
21271+ STBIR_NO_UNROLL(encode);
21272+ STBIR_CLAMP(f0, 0, 255);
21273+ output[0] = (unsigned char)f0;
21274+ #if stbir__coder_min_num >= 2
21275+ STBIR_CLAMP(f1, 0, 255);
21276+ output[1] = (unsigned char)f1;
21277+ #endif
21278+ #if stbir__coder_min_num >= 3
21279+ STBIR_CLAMP(f2, 0, 255);
21280+ output[2] = (unsigned char)f2;
21281+ #endif
21282+ output += stbir__coder_min_num;
21283+ encode += stbir__coder_min_num;
21284+ }
21285+ #endif
21286+ #endif
21287+}
21288+
21289+static float * STBIR__CODER_NAME(stbir__decode_uint8_linear)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
21290+{
21291+ float STBIR_STREAMOUT_PTR( * ) decode = decodep;
21292+ float * decode_end = (float*) decode + width_times_channels;
21293+ unsigned char const * input = (unsigned char const*)inputp;
21294+
21295+ #ifdef STBIR_SIMD
21296+ unsigned char const * end_input_m16 = input + width_times_channels - 16;
21297+ if ( width_times_channels >= 16 )
21298+ {
21299+ decode_end -= 16;
21300+ STBIR_NO_UNROLL_LOOP_START_INF_FOR
21301+ for(;;)
21302+ {
21303+ #if defined(STBIR_SIMD16)
21304+ stbir__simdi i; stbir__simdi16 o;
21305+ stbir__simdf16 of;
21306+ STBIR_NO_UNROLL(decode);
21307+ stbir__simdi_load( i, input );
21308+ stbir__decode_simdi_flip_u8( i );
21309+ stbir__simdi16_expand_u8_to_u32( o, i );
21310+ stbir__simdi16_convert_i32_to_float( of, o );
21311+ stbir__decode_simdf16_flip_i( of );
21312+ stbir__simdf16_store( decode, of );
21313+ #elif defined(STBIR_SIMD8)
21314+ stbir__simdi i; stbir__simdi8 o0,o1;
21315+ stbir__simdf8 of0, of1;
21316+ STBIR_NO_UNROLL(decode);
21317+ stbir__simdi_load( i, input );
21318+ stbir__decode_simdi_flip_u8( i );
21319+ stbir__simdi8_expand_u8_to_u32( o0, o1, i );
21320+ stbir__simdi8_convert_i32_to_float( of0, o0 );
21321+ stbir__simdi8_convert_i32_to_float( of1, o1 );
21322+ stbir__decode_simdf8_flip_i( of0 );
21323+ stbir__decode_simdf8_flip_i( of1 );
21324+ stbir__simdf8_store( decode + 0, of0 );
21325+ stbir__simdf8_store( decode + 8, of1 );
21326+ #else
21327+ stbir__simdi i, o0, o1, o2, o3;
21328+ stbir__simdf of0, of1, of2, of3;
21329+ STBIR_NO_UNROLL(decode);
21330+ stbir__simdi_load( i, input );
21331+ stbir__decode_simdi_flip_u8( i );
21332+ stbir__simdi_expand_u8_to_u32( o0,o1,o2,o3,i);
21333+ stbir__simdi_convert_i32_to_float( of0, o0 );
21334+ stbir__simdi_convert_i32_to_float( of1, o1 );
21335+ stbir__simdi_convert_i32_to_float( of2, o2 );
21336+ stbir__simdi_convert_i32_to_float( of3, o3 );
21337+ stbir__decode_simdf4_flip_i( of0 );
21338+ stbir__decode_simdf4_flip_i( of1 );
21339+ stbir__decode_simdf4_flip_i( of2 );
21340+ stbir__decode_simdf4_flip_i( of3 );
21341+ stbir__simdf_store( decode + 0, of0 );
21342+ stbir__simdf_store( decode + 4, of1 );
21343+ stbir__simdf_store( decode + 8, of2 );
21344+ stbir__simdf_store( decode + 12, of3 );
21345+#endif
21346+ decode += 16;
21347+ input += 16;
21348+ if ( decode <= decode_end )
21349+ continue;
21350+ if ( decode == ( decode_end + 16 ) )
21351+ break;
21352+ decode = decode_end; // backup and do last couple
21353+ input = end_input_m16;
21354+ }
21355+ return decode_end + 16;
21356+ }
21357+ #endif
21358+
21359+ // try to do blocks of 4 when you can
21360+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
21361+ decode += 4;
21362+ STBIR_SIMD_NO_UNROLL_LOOP_START
21363+ while( decode <= decode_end )
21364+ {
21365+ STBIR_SIMD_NO_UNROLL(decode);
21366+ decode[0-4] = ((float)(input[stbir__decode_order0]));
21367+ decode[1-4] = ((float)(input[stbir__decode_order1]));
21368+ decode[2-4] = ((float)(input[stbir__decode_order2]));
21369+ decode[3-4] = ((float)(input[stbir__decode_order3]));
21370+ decode += 4;
21371+ input += 4;
21372+ }
21373+ decode -= 4;
21374+ #endif
21375+
21376+ // do the remnants
21377+ #if stbir__coder_min_num < 4
21378+ STBIR_NO_UNROLL_LOOP_START
21379+ while( decode < decode_end )
21380+ {
21381+ STBIR_NO_UNROLL(decode);
21382+ decode[0] = ((float)(input[stbir__decode_order0]));
21383+ #if stbir__coder_min_num >= 2
21384+ decode[1] = ((float)(input[stbir__decode_order1]));
21385+ #endif
21386+ #if stbir__coder_min_num >= 3
21387+ decode[2] = ((float)(input[stbir__decode_order2]));
21388+ #endif
21389+ decode += stbir__coder_min_num;
21390+ input += stbir__coder_min_num;
21391+ }
21392+ #endif
21393+ return decode_end;
21394+}
21395+
21396+static void STBIR__CODER_NAME( stbir__encode_uint8_linear )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
21397+{
21398+ unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char *) outputp;
21399+ unsigned char * end_output = ( (unsigned char *) output ) + width_times_channels;
21400+
21401+ #ifdef STBIR_SIMD
21402+ // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
21403+ // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
21404+ STBIR_SIMD_NO_UNROLL_LOOP_START
21405+ while ( ( end_output - output ) >= stbir__simdfX_float_count*2 )
21406+ {
21407+ stbir__simdfX e0, e1;
21408+ STBIR_SIMD_NO_UNROLL(encode);
21409+ stbir__simdfX_add_mem( e0, STBIR_simd_point5X, encode );
21410+ stbir__simdfX_add_mem( e1, STBIR_simd_point5X, encode+stbir__simdfX_float_count );
21411+ stbir__encode_simdfX_unflip( e0 );
21412+ stbir__encode_simdfX_unflip( e1 );
21413+ #if defined(STBIR_SIMD16)
21414+ {
21415+ stbir__simdi8 i;
21416+ stbir__simdf16_pack_to_32bytes( i, e0, e1 );
21417+ stbir__simdi8_store( output, i );
21418+ }
21419+ #elif defined(STBIR_SIMD8)
21420+ {
21421+ stbir__simdi i;
21422+ stbir__simdf8_pack_to_16bytes( i, e0, e1 );
21423+ stbir__simdi_store( output, i );
21424+ }
21425+ #else
21426+ {
21427+ stbir__simdi i;
21428+ stbir__simdf_pack_to_8bytes( i, e0, e1 );
21429+ stbir__simdi_store2( output, i );
21430+ }
21431+ #endif
21432+ encode += stbir__simdfX_float_count*2;
21433+ output += stbir__simdfX_float_count*2;
21434+ }
21435+
21436+ // try to do blocks of 4 when you can
21437+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
21438+ output += 4;
21439+ STBIR_NO_UNROLL_LOOP_START
21440+ while( output <= end_output )
21441+ {
21442+ stbir__simdf e0;
21443+ stbir__simdi i0;
21444+ STBIR_NO_UNROLL(encode);
21445+ stbir__simdf_load( e0, encode );
21446+ stbir__simdf_add( e0, STBIR__CONSTF(STBIR_simd_point5), e0 );
21447+ stbir__encode_simdf4_unflip( e0 );
21448+ stbir__simdf_pack_to_8bytes( i0, e0, e0 ); // only use first 4
21449+ *(int*)(output-4) = stbir__simdi_to_int( i0 );
21450+ output += 4;
21451+ encode += 4;
21452+ }
21453+ output -= 4;
21454+ #endif
21455+
21456+ #else
21457+
21458+ // try to do blocks of 4 when you can
21459+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
21460+ output += 4;
21461+ while( output <= end_output )
21462+ {
21463+ float f0 = encode[stbir__encode_order0] + 0.5f;
21464+ float f1 = encode[stbir__encode_order1] + 0.5f;
21465+ float f2 = encode[stbir__encode_order2] + 0.5f;
21466+ float f3 = encode[stbir__encode_order3] + 0.5f;
21467+ STBIR_CLAMP(f0, 0, 255);
21468+ STBIR_CLAMP(f1, 0, 255);
21469+ STBIR_CLAMP(f2, 0, 255);
21470+ STBIR_CLAMP(f3, 0, 255);
21471+ output[0-4] = (unsigned char)f0;
21472+ output[1-4] = (unsigned char)f1;
21473+ output[2-4] = (unsigned char)f2;
21474+ output[3-4] = (unsigned char)f3;
21475+ output += 4;
21476+ encode += 4;
21477+ }
21478+ output -= 4;
21479+ #endif
21480+
21481+ #endif
21482+
21483+ // do the remnants
21484+ #if stbir__coder_min_num < 4
21485+ STBIR_NO_UNROLL_LOOP_START
21486+ while( output < end_output )
21487+ {
21488+ float f0 = encode[stbir__encode_order0] + 0.5f;
21489+ #if stbir__coder_min_num >= 2
21490+ float f1 = encode[stbir__encode_order1] + 0.5f;
21491+ #endif
21492+ #if stbir__coder_min_num >= 3
21493+ float f2 = encode[stbir__encode_order2] + 0.5f;
21494+ #endif
21495+ STBIR_NO_UNROLL(encode);
21496+ STBIR_CLAMP(f0, 0, 255);
21497+ output[0] = (unsigned char)f0;
21498+ #if stbir__coder_min_num >= 2
21499+ STBIR_CLAMP(f1, 0, 255);
21500+ output[1] = (unsigned char)f1;
21501+ #endif
21502+ #if stbir__coder_min_num >= 3
21503+ STBIR_CLAMP(f2, 0, 255);
21504+ output[2] = (unsigned char)f2;
21505+ #endif
21506+ output += stbir__coder_min_num;
21507+ encode += stbir__coder_min_num;
21508+ }
21509+ #endif
21510+}
21511+
21512+static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
21513+{
21514+ float STBIR_STREAMOUT_PTR( * ) decode = decodep;
21515+ float * decode_end = (float*) decode + width_times_channels;
21516+ unsigned char const * input = (unsigned char const *)inputp;
21517+
21518+ // try to do blocks of 4 when you can
21519+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
21520+ decode += 4;
21521+ while( decode <= decode_end )
21522+ {
21523+ decode[0-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order0 ] ];
21524+ decode[1-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order1 ] ];
21525+ decode[2-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order2 ] ];
21526+ decode[3-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order3 ] ];
21527+ decode += 4;
21528+ input += 4;
21529+ }
21530+ decode -= 4;
21531+ #endif
21532+
21533+ // do the remnants
21534+ #if stbir__coder_min_num < 4
21535+ STBIR_NO_UNROLL_LOOP_START
21536+ while( decode < decode_end )
21537+ {
21538+ STBIR_NO_UNROLL(decode);
21539+ decode[0] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order0 ] ];
21540+ #if stbir__coder_min_num >= 2
21541+ decode[1] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order1 ] ];
21542+ #endif
21543+ #if stbir__coder_min_num >= 3
21544+ decode[2] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order2 ] ];
21545+ #endif
21546+ decode += stbir__coder_min_num;
21547+ input += stbir__coder_min_num;
21548+ }
21549+ #endif
21550+ return decode_end;
21551+}
21552+
21553+#define stbir__min_max_shift20( i, f ) \
21554+ do { \
21555+ stbir__simdf_max( f, f, stbir_simdf_casti(STBIR__CONSTI( STBIR_almost_zero )) ); \
21556+ stbir__simdf_min( f, f, stbir_simdf_casti(STBIR__CONSTI( STBIR_almost_one )) ); \
21557+ stbir__simdi_32shr( i, stbir_simdi_castf( f ), 20 ); \
21558+ } while(0)
21559+
21560+#define stbir__scale_and_convert( i, f ) \
21561+ do { \
21562+ stbir__simdf_madd( f, STBIR__CONSTF( STBIR_simd_point5 ), STBIR__CONSTF( STBIR_max_uint8_as_float ), f ); \
21563+ stbir__simdf_max( f, f, stbir__simdf_zeroP() ); \
21564+ stbir__simdf_min( f, f, STBIR__CONSTF( STBIR_max_uint8_as_float ) ); \
21565+ stbir__simdf_convert_float_to_i32( i, f ); \
21566+ } while(0)
21567+
21568+#define stbir__linear_to_srgb_finish( i, f ) \
21569+{ \
21570+ stbir__simdi temp; \
21571+ stbir__simdi_32shr( temp, stbir_simdi_castf( f ), 12 ) ; \
21572+ stbir__simdi_and( temp, temp, STBIR__CONSTI(STBIR_mantissa_mask) ); \
21573+ stbir__simdi_or( temp, temp, STBIR__CONSTI(STBIR_topscale) ); \
21574+ stbir__simdi_16madd( i, i, temp ); \
21575+ stbir__simdi_32shr( i, i, 16 ); \
21576+}
21577+
21578+// look up sRGB bytes for 4 float lanes per register (indexed by the float bits shifted right 20, see fp32_to_srgb8_tab4)
21579+#define stbir__simdi_srgb_lookup_one( v, temp ) \
21580+ do { \
21581+ temp.m128i_i128 = v; \
21582+ temp.m128i_u32[0] = STBIR__SRGB8_TAB4( temp.m128i_i32[0] ); \
21583+ temp.m128i_u32[1] = STBIR__SRGB8_TAB4( temp.m128i_i32[1] ); \
21584+ temp.m128i_u32[2] = STBIR__SRGB8_TAB4( temp.m128i_i32[2] ); \
21585+ temp.m128i_u32[3] = STBIR__SRGB8_TAB4( temp.m128i_i32[3] ); \
21586+ v = temp.m128i_i128; \
21587+ } while(0)
21588+
21589+#define stbir__simdi_srgb_lookup2( v0,v1 ) \
21590+ do { \
21591+ stbir__simdi_u32 temp0,temp1; \
21592+ stbir__simdi_srgb_lookup_one( v0, temp0 ); \
21593+ stbir__simdi_srgb_lookup_one( v1, temp1 ); \
21594+ } while(0)
21595+
21596+#define stbir__simdi_srgb_lookup3( v0,v1,v2 ) \
21597+ do { \
21598+ stbir__simdi_u32 temp0,temp1,temp2; \
21599+ stbir__simdi_srgb_lookup_one( v0, temp0 ); \
21600+ stbir__simdi_srgb_lookup_one( v1, temp1 ); \
21601+ stbir__simdi_srgb_lookup_one( v2, temp2 ); \
21602+ } while(0)
21603+
21604+#define stbir__simdi_srgb_lookup4( v0,v1,v2,v3 ) \
21605+ do { \
21606+ stbir__simdi_u32 temp0,temp1,temp2,temp3; \
21607+ stbir__simdi_srgb_lookup_one( v0, temp0 ); \
21608+ stbir__simdi_srgb_lookup_one( v1, temp1 ); \
21609+ stbir__simdi_srgb_lookup_one( v2, temp2 ); \
21610+ stbir__simdi_srgb_lookup_one( v3, temp3 ); \
21611+ } while(0)
21612+
21613+static void STBIR__CODER_NAME( stbir__encode_uint8_srgb )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
21614+{
21615+ unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
21616+ unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
21617+
21618+ #ifdef STBIR_SIMD
21619+
21620+ // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
21621+ // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
21622+ STBIR_SIMD_NO_UNROLL_LOOP_START
21623+ while ( ( end_output - output ) >= 16 )
21624+ {
21625+ stbir__simdf f0, f1, f2, f3;
21626+ stbir__simdi i0, i1, i2, i3;
21627+ STBIR_SIMD_NO_UNROLL(encode);
21628+
21629+ stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
21630+
21631+ stbir__min_max_shift20( i0, f0 );
21632+ stbir__min_max_shift20( i1, f1 );
21633+ stbir__min_max_shift20( i2, f2 );
21634+ stbir__min_max_shift20( i3, f3 );
21635+
21636+ stbir__simdi_srgb_lookup4( i0, i1, i2, i3 );
21637+
21638+ stbir__linear_to_srgb_finish( i0, f0 );
21639+ stbir__linear_to_srgb_finish( i1, f1 );
21640+ stbir__linear_to_srgb_finish( i2, f2 );
21641+ stbir__linear_to_srgb_finish( i3, f3 );
21642+
21643+ stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
21644+
21645+ encode += 16;
21646+ output += 16;
21647+ }
21648+ #endif
21649+
21650+ // try to do blocks of 4 when you can
21651+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
21652+ output += 4;
21653+ STBIR_SIMD_NO_UNROLL_LOOP_START
21654+ while ( output <= end_output )
21655+ {
21656+ float e0, e1, e2, e3;
21657+ STBIR_SIMD_NO_UNROLL(encode);
21658+
21659+ e0 = encode[stbir__encode_order0];
21660+ e1 = encode[stbir__encode_order1];
21661+ e2 = encode[stbir__encode_order2];
21662+ e3 = encode[stbir__encode_order3];
21663+ output[0-4] = stbir__linear_to_srgb_uchar( e0 );
21664+ output[1-4] = stbir__linear_to_srgb_uchar( e1 );
21665+ output[2-4] = stbir__linear_to_srgb_uchar( e2 );
21666+ output[3-4] = stbir__linear_to_srgb_uchar( e3 );
21667+
21668+ output += 4;
21669+ encode += 4;
21670+ }
21671+ output -= 4;
21672+ #endif
21673+
21674+ // do the remnants
21675+ #if stbir__coder_min_num < 4
21676+ STBIR_NO_UNROLL_LOOP_START
21677+ while( output < end_output )
21678+ {
21679+ float e0;
21680+ #if stbir__coder_min_num >= 2
21681+ float e1;
21682+ #endif
21683+ #if stbir__coder_min_num >= 3
21684+ float e2;
21685+ #endif
21686+ STBIR_NO_UNROLL(encode);
21687+ e0 = encode[stbir__encode_order0];
21688+ #if stbir__coder_min_num >= 2
21689+ e1 = encode[stbir__encode_order1];
21690+ #endif
21691+ #if stbir__coder_min_num >= 3
21692+ e2 = encode[stbir__encode_order2];
21693+ #endif
21694+ output[0] = stbir__linear_to_srgb_uchar( e0 );
21695+ #if stbir__coder_min_num >= 2
21696+ output[1] = stbir__linear_to_srgb_uchar( e1 );
21697+ #endif
21698+ #if stbir__coder_min_num >= 3
21699+ output[2] = stbir__linear_to_srgb_uchar( e2 );
21700+ #endif
21701+ output += stbir__coder_min_num;
21702+ encode += stbir__coder_min_num;
21703+ }
21704+ #endif
21705+}
21706+
21707+#if ( stbir__coder_min_num == 4 ) || ( ( stbir__coder_min_num == 1 ) && ( !defined(stbir__decode_swizzle) ) )
21708+
21709+static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb4_linearalpha)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
21710+{
21711+ float STBIR_STREAMOUT_PTR( * ) decode = decodep;
21712+ float * decode_end = (float*) decode + width_times_channels;
21713+ unsigned char const * input = (unsigned char const *)inputp;
21714+
21715+ do {
21716+ decode[0] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
21717+ decode[1] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order1] ];
21718+ decode[2] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order2] ];
21719+ decode[3] = ( (float) input[stbir__decode_order3] ) * stbir__max_uint8_as_float_inverted;
21720+ input += 4;
21721+ decode += 4;
21722+ } while( decode < decode_end );
21723+ return decode_end;
21724+}
21725+
21726+
21727+static void STBIR__CODER_NAME( stbir__encode_uint8_srgb4_linearalpha )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
21728+{
21729+ unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
21730+ unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
21731+
21732+ #ifdef STBIR_SIMD
21733+
21734+ // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
21735+ // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
21736+ STBIR_SIMD_NO_UNROLL_LOOP_START
21737+ while ( ( end_output - output ) >= 16 )
21738+ {
21739+ stbir__simdf f0, f1, f2, f3;
21740+ stbir__simdi i0, i1, i2, i3;
21741+
21742+ STBIR_SIMD_NO_UNROLL(encode);
21743+ stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
21744+
21745+ stbir__min_max_shift20( i0, f0 );
21746+ stbir__min_max_shift20( i1, f1 );
21747+ stbir__min_max_shift20( i2, f2 );
21748+ stbir__scale_and_convert( i3, f3 );
21749+
21750+ stbir__simdi_srgb_lookup3( i0, i1, i2 );
21751+
21752+ stbir__linear_to_srgb_finish( i0, f0 );
21753+ stbir__linear_to_srgb_finish( i1, f1 );
21754+ stbir__linear_to_srgb_finish( i2, f2 );
21755+
21756+ stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
21757+
21758+ output += 16;
21759+ encode += 16;
21760+
21761+ }
21762+ #endif
21763+
21764+ // the SIMD loop above can have finished the row, so this has to test first
21765+ STBIR_SIMD_NO_UNROLL_LOOP_START
21766+ while ( output < end_output )
21767+ {
21768+ float f;
21769+ STBIR_SIMD_NO_UNROLL(encode);
21770+
21771+ output[stbir__decode_order0] = stbir__linear_to_srgb_uchar( encode[0] );
21772+ output[stbir__decode_order1] = stbir__linear_to_srgb_uchar( encode[1] );
21773+ output[stbir__decode_order2] = stbir__linear_to_srgb_uchar( encode[2] );
21774+
21775+ f = encode[3] * stbir__max_uint8_as_float + 0.5f;
21776+ STBIR_CLAMP(f, 0, 255);
21777+ output[stbir__decode_order3] = (unsigned char) f;
21778+
21779+ output += 4;
21780+ encode += 4;
21781+ }
21782+}
21783+
21784+#endif
21785+
21786+#if ( stbir__coder_min_num == 2 ) || ( ( stbir__coder_min_num == 1 ) && ( !defined(stbir__decode_swizzle) ) )
21787+
21788+static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb2_linearalpha)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
21789+{
21790+ float STBIR_STREAMOUT_PTR( * ) decode = decodep;
21791+ float * decode_end = (float*) decode + width_times_channels;
21792+ unsigned char const * input = (unsigned char const *)inputp;
21793+
21794+ decode += 4;
21795+ while( decode <= decode_end )
21796+ {
21797+ decode[0-4] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
21798+ decode[1-4] = ( (float) input[stbir__decode_order1] ) * stbir__max_uint8_as_float_inverted;
21799+ decode[2-4] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0+2] ];
21800+ decode[3-4] = ( (float) input[stbir__decode_order1+2] ) * stbir__max_uint8_as_float_inverted;
21801+ input += 4;
21802+ decode += 4;
21803+ }
21804+ decode -= 4;
21805+ if( decode < decode_end )
21806+ {
21807+ decode[0] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
21808+ decode[1] = ( (float) input[stbir__decode_order1] ) * stbir__max_uint8_as_float_inverted;
21809+ }
21810+ return decode_end;
21811+}
21812+
21813+static void STBIR__CODER_NAME( stbir__encode_uint8_srgb2_linearalpha )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
21814+{
21815+ unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
21816+ unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
21817+
21818+ #ifdef STBIR_SIMD
21819+
21820+ // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
21821+ // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
21822+ STBIR_SIMD_NO_UNROLL_LOOP_START
21823+ while ( ( end_output - output ) >= 16 )
21824+ {
21825+ stbir__simdf f0, f1, f2, f3;
21826+ stbir__simdi i0, i1, i2, i3;
21827+
21828+ STBIR_SIMD_NO_UNROLL(encode);
21829+ stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
21830+
21831+ stbir__min_max_shift20( i0, f0 );
21832+ stbir__scale_and_convert( i1, f1 );
21833+ stbir__min_max_shift20( i2, f2 );
21834+ stbir__scale_and_convert( i3, f3 );
21835+
21836+ stbir__simdi_srgb_lookup2( i0, i2 );
21837+
21838+ stbir__linear_to_srgb_finish( i0, f0 );
21839+ stbir__linear_to_srgb_finish( i2, f2 );
21840+
21841+ stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
21842+
21843+ output += 16;
21844+ encode += 16;
21845+ }
21846+ #endif
21847+
21848+ // the SIMD loop above can have finished the row, so this has to test first
21849+ STBIR_SIMD_NO_UNROLL_LOOP_START
21850+ while ( output < end_output )
21851+ {
21852+ float f;
21853+ STBIR_SIMD_NO_UNROLL(encode);
21854+
21855+ output[stbir__decode_order0] = stbir__linear_to_srgb_uchar( encode[0] );
21856+
21857+ f = encode[1] * stbir__max_uint8_as_float + 0.5f;
21858+ STBIR_CLAMP(f, 0, 255);
21859+ output[stbir__decode_order1] = (unsigned char) f;
21860+
21861+ output += 2;
21862+ encode += 2;
21863+ }
21864+}
21865+
21866+#endif
21867+
21868+static float * STBIR__CODER_NAME(stbir__decode_uint16_linear_scaled)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
21869+{
21870+ float STBIR_STREAMOUT_PTR( * ) decode = decodep;
21871+ float * decode_end = (float*) decode + width_times_channels;
21872+ unsigned short const * input = (unsigned short const *)inputp;
21873+
21874+ #ifdef STBIR_SIMD
21875+ // a block is 16 words at 16 wide (one 32 byte load), 8 otherwise
21876+ unsigned short const * end_input_mb = input + width_times_channels - STBIR__U16_DECODE_BLOCK;
21877+ if ( width_times_channels >= STBIR__U16_DECODE_BLOCK )
21878+ {
21879+ decode_end -= STBIR__U16_DECODE_BLOCK;
21880+ STBIR_NO_UNROLL_LOOP_START_INF_FOR
21881+ for(;;)
21882+ {
21883+ #if defined(STBIR_SIMD16)
21884+ stbir__simdi8 i; stbir__simdi16 o;
21885+ stbir__simdf16 of;
21886+ STBIR_NO_UNROLL(decode);
21887+ stbir__simdi8_load( i, input );
21888+ stbir__decode_simdi8_flip_u16( i );
21889+ stbir__simdi16_expand_u16_to_u32( o, i );
21890+ stbir__simdi16_convert_i32_to_float( of, o );
21891+ stbir__simdf16_mult( of, of, STBIR_max_uint16_as_float_inverted16 );
21892+ stbir__decode_simdf16_flip_i( of );
21893+ stbir__simdf16_store( decode, of );
21894+ #elif defined(STBIR_SIMD8)
21895+ stbir__simdi i; stbir__simdi8 o;
21896+ stbir__simdf8 of;
21897+ STBIR_NO_UNROLL(decode);
21898+ stbir__simdi_load( i, input );
21899+ stbir__decode_simdi_flip_u16( i );
21900+ stbir__simdi8_expand_u16_to_u32( o, i );
21901+ stbir__simdi8_convert_i32_to_float( of, o );
21902+ stbir__simdf8_mult( of, of, STBIR_max_uint16_as_float_inverted8);
21903+ stbir__decode_simdf8_flip_i( of );
21904+ stbir__simdf8_store( decode + 0, of );
21905+ #else
21906+ stbir__simdi i, o0, o1;
21907+ stbir__simdf of0, of1;
21908+ STBIR_NO_UNROLL(decode);
21909+ stbir__simdi_load( i, input );
21910+ stbir__decode_simdi_flip_u16( i );
21911+ stbir__simdi_expand_u16_to_u32( o0,o1,i );
21912+ stbir__simdi_convert_i32_to_float( of0, o0 );
21913+ stbir__simdi_convert_i32_to_float( of1, o1 );
21914+ stbir__simdf_mult( of0, of0, STBIR__CONSTF(STBIR_max_uint16_as_float_inverted) );
21915+ stbir__simdf_mult( of1, of1, STBIR__CONSTF(STBIR_max_uint16_as_float_inverted));
21916+ stbir__decode_simdf4_flip_i( of0 );
21917+ stbir__decode_simdf4_flip_i( of1 );
21918+ stbir__simdf_store( decode + 0, of0 );
21919+ stbir__simdf_store( decode + 4, of1 );
21920+ #endif
21921+ decode += STBIR__U16_DECODE_BLOCK;
21922+ input += STBIR__U16_DECODE_BLOCK;
21923+ if ( decode <= decode_end )
21924+ continue;
21925+ if ( decode == ( decode_end + STBIR__U16_DECODE_BLOCK ) )
21926+ break;
21927+ decode = decode_end; // backup and do last couple
21928+ input = end_input_mb;
21929+ }
21930+ return decode_end + STBIR__U16_DECODE_BLOCK;
21931+ }
21932+ #endif
21933+
21934+ // try to do blocks of 4 when you can
21935+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
21936+ decode += 4;
21937+ STBIR_SIMD_NO_UNROLL_LOOP_START
21938+ while( decode <= decode_end )
21939+ {
21940+ STBIR_SIMD_NO_UNROLL(decode);
21941+ decode[0-4] = ((float)(input[stbir__decode_order0])) * stbir__max_uint16_as_float_inverted;
21942+ decode[1-4] = ((float)(input[stbir__decode_order1])) * stbir__max_uint16_as_float_inverted;
21943+ decode[2-4] = ((float)(input[stbir__decode_order2])) * stbir__max_uint16_as_float_inverted;
21944+ decode[3-4] = ((float)(input[stbir__decode_order3])) * stbir__max_uint16_as_float_inverted;
21945+ decode += 4;
21946+ input += 4;
21947+ }
21948+ decode -= 4;
21949+ #endif
21950+
21951+ // do the remnants
21952+ #if stbir__coder_min_num < 4
21953+ STBIR_NO_UNROLL_LOOP_START
21954+ while( decode < decode_end )
21955+ {
21956+ STBIR_NO_UNROLL(decode);
21957+ decode[0] = ((float)(input[stbir__decode_order0])) * stbir__max_uint16_as_float_inverted;
21958+ #if stbir__coder_min_num >= 2
21959+ decode[1] = ((float)(input[stbir__decode_order1])) * stbir__max_uint16_as_float_inverted;
21960+ #endif
21961+ #if stbir__coder_min_num >= 3
21962+ decode[2] = ((float)(input[stbir__decode_order2])) * stbir__max_uint16_as_float_inverted;
21963+ #endif
21964+ decode += stbir__coder_min_num;
21965+ input += stbir__coder_min_num;
21966+ }
21967+ #endif
21968+ return decode_end;
21969+}
21970+
21971+
21972+static void STBIR__CODER_NAME(stbir__encode_uint16_linear_scaled)( void * outputp, ptrdiff_t width_times_channels, float const * encode )
21973+{
21974+ unsigned short STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned short*) outputp;
21975+ unsigned short * end_output = ( (unsigned short*) output ) + width_times_channels;
21976+
21977+ #ifdef STBIR_SIMD
21978+ {
21979+ // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
21980+ // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
21981+ STBIR_SIMD_NO_UNROLL_LOOP_START
21982+ while ( ( end_output - output ) >= stbir__simdfX_float_count*2 )
21983+ {
21984+ stbir__simdfX e0, e1;
21985+ stbir__simdiX i;
21986+ STBIR_SIMD_NO_UNROLL(encode);
21987+ stbir__simdfX_madd_mem( e0, STBIR_simd_point5X, STBIR_max_uint16_as_floatX, encode );
21988+ stbir__simdfX_madd_mem( e1, STBIR_simd_point5X, STBIR_max_uint16_as_floatX, encode+stbir__simdfX_float_count );
21989+ stbir__encode_simdfX_unflip( e0 );
21990+ stbir__encode_simdfX_unflip( e1 );
21991+ stbir__simdfX_pack_to_words( i, e0, e1 );
21992+ stbir__simdiX_store( output, i );
21993+ encode += stbir__simdfX_float_count*2;
21994+ output += stbir__simdfX_float_count*2;
21995+ }
21996+ }
21997+
21998+ // try to do blocks of 4 when you can
21999+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
22000+ output += 4;
22001+ STBIR_NO_UNROLL_LOOP_START
22002+ while( output <= end_output )
22003+ {
22004+ stbir__simdf e;
22005+ stbir__simdi i;
22006+ STBIR_NO_UNROLL(encode);
22007+ stbir__simdf_load( e, encode );
22008+ stbir__simdf_madd( e, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), e );
22009+ stbir__encode_simdf4_unflip( e );
22010+ stbir__simdf_pack_to_8words( i, e, e ); // only use first 4
22011+ stbir__simdi_store2( output-4, i );
22012+ output += 4;
22013+ encode += 4;
22014+ }
22015+ output -= 4;
22016+ #endif
22017+
22018+ // do the remnants
22019+ #if stbir__coder_min_num < 4
22020+ STBIR_NO_UNROLL_LOOP_START
22021+ while( output < end_output )
22022+ {
22023+ // read the whole pixel before writing any of it - the encode can be in place, and a swizzle reads a channel after
22024+ // the one it lands on has been overwritten
22025+ stbir__simdf e0;
22026+ #if stbir__coder_min_num >= 2
22027+ stbir__simdf e1;
22028+ #endif
22029+ #if stbir__coder_min_num >= 3
22030+ stbir__simdf e2;
22031+ #endif
22032+ STBIR_NO_UNROLL(encode);
22033+ stbir__simdf_madd1_mem( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order0 );
22034+ #if stbir__coder_min_num >= 2
22035+ stbir__simdf_madd1_mem( e1, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order1 );
22036+ #endif
22037+ #if stbir__coder_min_num >= 3
22038+ stbir__simdf_madd1_mem( e2, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order2 );
22039+ #endif
22040+ output[0] = stbir__simdf_convert_float_to_short( e0 );
22041+ #if stbir__coder_min_num >= 2
22042+ output[1] = stbir__simdf_convert_float_to_short( e1 );
22043+ #endif
22044+ #if stbir__coder_min_num >= 3
22045+ output[2] = stbir__simdf_convert_float_to_short( e2 );
22046+ #endif
22047+ output += stbir__coder_min_num;
22048+ encode += stbir__coder_min_num;
22049+ }
22050+ #endif
22051+
22052+ #else
22053+
22054+ // try to do blocks of 4 when you can
22055+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
22056+ output += 4;
22057+ STBIR_SIMD_NO_UNROLL_LOOP_START
22058+ while( output <= end_output )
22059+ {
22060+ // read the whole pixel before writing any of it - the encode can be in place, and a swizzle reads a channel after
22061+ // the one it lands on has been overwritten
22062+ float f0 = encode[stbir__encode_order0] * stbir__max_uint16_as_float + 0.5f;
22063+ float f1 = encode[stbir__encode_order1] * stbir__max_uint16_as_float + 0.5f;
22064+ float f2 = encode[stbir__encode_order2] * stbir__max_uint16_as_float + 0.5f;
22065+ float f3 = encode[stbir__encode_order3] * stbir__max_uint16_as_float + 0.5f;
22066+ STBIR_SIMD_NO_UNROLL(encode);
22067+ STBIR_CLAMP(f0, 0, 65535); output[0-4] = (unsigned short)f0;
22068+ STBIR_CLAMP(f1, 0, 65535); output[1-4] = (unsigned short)f1;
22069+ STBIR_CLAMP(f2, 0, 65535); output[2-4] = (unsigned short)f2;
22070+ STBIR_CLAMP(f3, 0, 65535); output[3-4] = (unsigned short)f3;
22071+ output += 4;
22072+ encode += 4;
22073+ }
22074+ output -= 4;
22075+ #endif
22076+
22077+ // do the remnants
22078+ #if stbir__coder_min_num < 4
22079+ STBIR_NO_UNROLL_LOOP_START
22080+ while( output < end_output )
22081+ {
22082+ float f0 = encode[stbir__encode_order0] * stbir__max_uint16_as_float + 0.5f;
22083+ #if stbir__coder_min_num >= 2
22084+ float f1 = encode[stbir__encode_order1] * stbir__max_uint16_as_float + 0.5f;
22085+ #endif
22086+ #if stbir__coder_min_num >= 3
22087+ float f2 = encode[stbir__encode_order2] * stbir__max_uint16_as_float + 0.5f;
22088+ #endif
22089+ STBIR_NO_UNROLL(encode);
22090+ STBIR_CLAMP(f0, 0, 65535); output[0] = (unsigned short)f0;
22091+ #if stbir__coder_min_num >= 2
22092+ STBIR_CLAMP(f1, 0, 65535); output[1] = (unsigned short)f1;
22093+ #endif
22094+ #if stbir__coder_min_num >= 3
22095+ STBIR_CLAMP(f2, 0, 65535); output[2] = (unsigned short)f2;
22096+ #endif
22097+ output += stbir__coder_min_num;
22098+ encode += stbir__coder_min_num;
22099+ }
22100+ #endif
22101+ #endif
22102+}
22103+
22104+static float * STBIR__CODER_NAME(stbir__decode_uint16_linear)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
22105+{
22106+ float STBIR_STREAMOUT_PTR( * ) decode = decodep;
22107+ float * decode_end = (float*) decode + width_times_channels;
22108+ unsigned short const * input = (unsigned short const *)inputp;
22109+
22110+ #ifdef STBIR_SIMD
22111+ unsigned short const * end_input_mb = input + width_times_channels - STBIR__U16_DECODE_BLOCK;
22112+ if ( width_times_channels >= STBIR__U16_DECODE_BLOCK )
22113+ {
22114+ decode_end -= STBIR__U16_DECODE_BLOCK;
22115+ STBIR_NO_UNROLL_LOOP_START_INF_FOR
22116+ for(;;)
22117+ {
22118+ #if defined(STBIR_SIMD16)
22119+ stbir__simdi8 i; stbir__simdi16 o;
22120+ stbir__simdf16 of;
22121+ STBIR_NO_UNROLL(decode);
22122+ stbir__simdi8_load( i, input );
22123+ stbir__decode_simdi8_flip_u16( i );
22124+ stbir__simdi16_expand_u16_to_u32( o, i );
22125+ stbir__simdi16_convert_i32_to_float( of, o );
22126+ stbir__decode_simdf16_flip_i( of );
22127+ stbir__simdf16_store( decode, of );
22128+ #elif defined(STBIR_SIMD8)
22129+ stbir__simdi i; stbir__simdi8 o;
22130+ stbir__simdf8 of;
22131+ STBIR_NO_UNROLL(decode);
22132+ stbir__simdi_load( i, input );
22133+ stbir__decode_simdi_flip_u16( i );
22134+ stbir__simdi8_expand_u16_to_u32( o, i );
22135+ stbir__simdi8_convert_i32_to_float( of, o );
22136+ stbir__decode_simdf8_flip_i( of );
22137+ stbir__simdf8_store( decode + 0, of );
22138+ #else
22139+ stbir__simdi i, o0, o1;
22140+ stbir__simdf of0, of1;
22141+ STBIR_NO_UNROLL(decode);
22142+ stbir__simdi_load( i, input );
22143+ stbir__decode_simdi_flip_u16( i );
22144+ stbir__simdi_expand_u16_to_u32( o0, o1, i );
22145+ stbir__simdi_convert_i32_to_float( of0, o0 );
22146+ stbir__simdi_convert_i32_to_float( of1, o1 );
22147+ stbir__decode_simdf4_flip_i( of0 );
22148+ stbir__decode_simdf4_flip_i( of1 );
22149+ stbir__simdf_store( decode + 0, of0 );
22150+ stbir__simdf_store( decode + 4, of1 );
22151+ #endif
22152+ decode += STBIR__U16_DECODE_BLOCK;
22153+ input += STBIR__U16_DECODE_BLOCK;
22154+ if ( decode <= decode_end )
22155+ continue;
22156+ if ( decode == ( decode_end + STBIR__U16_DECODE_BLOCK ) )
22157+ break;
22158+ decode = decode_end; // backup and do last couple
22159+ input = end_input_mb;
22160+ }
22161+ return decode_end + STBIR__U16_DECODE_BLOCK;
22162+ }
22163+ #endif
22164+
22165+ // try to do blocks of 4 when you can
22166+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
22167+ decode += 4;
22168+ STBIR_SIMD_NO_UNROLL_LOOP_START
22169+ while( decode <= decode_end )
22170+ {
22171+ STBIR_SIMD_NO_UNROLL(decode);
22172+ decode[0-4] = ((float)(input[stbir__decode_order0]));
22173+ decode[1-4] = ((float)(input[stbir__decode_order1]));
22174+ decode[2-4] = ((float)(input[stbir__decode_order2]));
22175+ decode[3-4] = ((float)(input[stbir__decode_order3]));
22176+ decode += 4;
22177+ input += 4;
22178+ }
22179+ decode -= 4;
22180+ #endif
22181+
22182+ // do the remnants
22183+ #if stbir__coder_min_num < 4
22184+ STBIR_NO_UNROLL_LOOP_START
22185+ while( decode < decode_end )
22186+ {
22187+ STBIR_NO_UNROLL(decode);
22188+ decode[0] = ((float)(input[stbir__decode_order0]));
22189+ #if stbir__coder_min_num >= 2
22190+ decode[1] = ((float)(input[stbir__decode_order1]));
22191+ #endif
22192+ #if stbir__coder_min_num >= 3
22193+ decode[2] = ((float)(input[stbir__decode_order2]));
22194+ #endif
22195+ decode += stbir__coder_min_num;
22196+ input += stbir__coder_min_num;
22197+ }
22198+ #endif
22199+ return decode_end;
22200+}
22201+
22202+static void STBIR__CODER_NAME(stbir__encode_uint16_linear)( void * outputp, ptrdiff_t width_times_channels, float const * encode )
22203+{
22204+ unsigned short STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned short*) outputp;
22205+ unsigned short * end_output = ( (unsigned short*) output ) + width_times_channels;
22206+
22207+ #ifdef STBIR_SIMD
22208+ {
22209+ // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
22210+ // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
22211+ STBIR_SIMD_NO_UNROLL_LOOP_START
22212+ while ( ( end_output - output ) >= stbir__simdfX_float_count*2 )
22213+ {
22214+ stbir__simdfX e0, e1;
22215+ stbir__simdiX i;
22216+ STBIR_SIMD_NO_UNROLL(encode);
22217+ stbir__simdfX_add_mem( e0, STBIR_simd_point5X, encode );
22218+ stbir__simdfX_add_mem( e1, STBIR_simd_point5X, encode+stbir__simdfX_float_count );
22219+ stbir__encode_simdfX_unflip( e0 );
22220+ stbir__encode_simdfX_unflip( e1 );
22221+ stbir__simdfX_pack_to_words( i, e0, e1 );
22222+ stbir__simdiX_store( output, i );
22223+ encode += stbir__simdfX_float_count*2;
22224+ output += stbir__simdfX_float_count*2;
22225+ }
22226+ }
22227+
22228+ // try to do blocks of 4 when you can
22229+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
22230+ output += 4;
22231+ STBIR_NO_UNROLL_LOOP_START
22232+ while( output <= end_output )
22233+ {
22234+ stbir__simdf e;
22235+ stbir__simdi i;
22236+ STBIR_NO_UNROLL(encode);
22237+ stbir__simdf_load( e, encode );
22238+ stbir__simdf_add( e, STBIR__CONSTF(STBIR_simd_point5), e );
22239+ stbir__encode_simdf4_unflip( e );
22240+ stbir__simdf_pack_to_8words( i, e, e ); // only use first 4
22241+ stbir__simdi_store2( output-4, i );
22242+ output += 4;
22243+ encode += 4;
22244+ }
22245+ output -= 4;
22246+ #endif
22247+
22248+ #else
22249+
22250+ // try to do blocks of 4 when you can
22251+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
22252+ output += 4;
22253+ STBIR_SIMD_NO_UNROLL_LOOP_START
22254+ while( output <= end_output )
22255+ {
22256+ // read the whole pixel before writing any of it - the encode can be in place, and a swizzle reads a channel after
22257+ // the one it lands on has been overwritten
22258+ float f0 = encode[stbir__encode_order0] + 0.5f;
22259+ float f1 = encode[stbir__encode_order1] + 0.5f;
22260+ float f2 = encode[stbir__encode_order2] + 0.5f;
22261+ float f3 = encode[stbir__encode_order3] + 0.5f;
22262+ STBIR_SIMD_NO_UNROLL(encode);
22263+ STBIR_CLAMP(f0, 0, 65535); output[0-4] = (unsigned short)f0;
22264+ STBIR_CLAMP(f1, 0, 65535); output[1-4] = (unsigned short)f1;
22265+ STBIR_CLAMP(f2, 0, 65535); output[2-4] = (unsigned short)f2;
22266+ STBIR_CLAMP(f3, 0, 65535); output[3-4] = (unsigned short)f3;
22267+ output += 4;
22268+ encode += 4;
22269+ }
22270+ output -= 4;
22271+ #endif
22272+
22273+ #endif
22274+
22275+ // do the remnants
22276+ #if stbir__coder_min_num < 4
22277+ STBIR_NO_UNROLL_LOOP_START
22278+ while( output < end_output )
22279+ {
22280+ float f0 = encode[stbir__encode_order0] + 0.5f;
22281+ #if stbir__coder_min_num >= 2
22282+ float f1 = encode[stbir__encode_order1] + 0.5f;
22283+ #endif
22284+ #if stbir__coder_min_num >= 3
22285+ float f2 = encode[stbir__encode_order2] + 0.5f;
22286+ #endif
22287+ STBIR_NO_UNROLL(encode);
22288+ STBIR_CLAMP(f0, 0, 65535); output[0] = (unsigned short)f0;
22289+ #if stbir__coder_min_num >= 2
22290+ STBIR_CLAMP(f1, 0, 65535); output[1] = (unsigned short)f1;
22291+ #endif
22292+ #if stbir__coder_min_num >= 3
22293+ STBIR_CLAMP(f2, 0, 65535); output[2] = (unsigned short)f2;
22294+ #endif
22295+ output += stbir__coder_min_num;
22296+ encode += stbir__coder_min_num;
22297+ }
22298+ #endif
22299+}
22300+
22301+static float * STBIR__CODER_NAME(stbir__decode_half_float_linear)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
22302+{
22303+ float STBIR_STREAMOUT_PTR( * ) decode = decodep;
22304+ float * decode_end = (float*) decode + width_times_channels;
22305+ stbir__FP16 const * input = (stbir__FP16 const *)inputp;
22306+
22307+ #ifdef STBIR_SIMD
22308+ if ( width_times_channels >= 8 )
22309+ {
22310+ stbir__FP16 const * end_input_m8 = input + width_times_channels - 8;
22311+ decode_end -= 8;
22312+ STBIR_NO_UNROLL_LOOP_START_INF_FOR
22313+ for(;;)
22314+ {
22315+ STBIR_NO_UNROLL(decode);
22316+
22317+ stbir__half_to_float_SIMD( decode, input );
22318+ #ifdef stbir__decode_swizzle
22319+ #ifdef STBIR_SIMD8
22320+ {
22321+ stbir__simdf8 of;
22322+ stbir__simdf8_load( of, decode );
22323+ stbir__decode_simdf8_flip( of );
22324+ stbir__simdf8_store( decode, of );
22325+ }
22326+ #else
22327+ {
22328+ stbir__simdf of0,of1;
22329+ stbir__simdf_load( of0, decode );
22330+ stbir__simdf_load( of1, decode+4 );
22331+ stbir__decode_simdf4_flip( of0 );
22332+ stbir__decode_simdf4_flip( of1 );
22333+ stbir__simdf_store( decode, of0 );
22334+ stbir__simdf_store( decode+4, of1 );
22335+ }
22336+ #endif
22337+ #endif
22338+ decode += 8;
22339+ input += 8;
22340+ if ( decode <= decode_end )
22341+ continue;
22342+ if ( decode == ( decode_end + 8 ) )
22343+ break;
22344+ decode = decode_end; // backup and do last couple
22345+ input = end_input_m8;
22346+ }
22347+ return decode_end + 8;
22348+ }
22349+ #endif
22350+
22351+ // try to do blocks of 4 when you can
22352+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
22353+ decode += 4;
22354+ STBIR_SIMD_NO_UNROLL_LOOP_START
22355+ while( decode <= decode_end )
22356+ {
22357+ STBIR_SIMD_NO_UNROLL(decode);
22358+ decode[0-4] = stbir__half_to_float(input[stbir__decode_order0]);
22359+ decode[1-4] = stbir__half_to_float(input[stbir__decode_order1]);
22360+ decode[2-4] = stbir__half_to_float(input[stbir__decode_order2]);
22361+ decode[3-4] = stbir__half_to_float(input[stbir__decode_order3]);
22362+ decode += 4;
22363+ input += 4;
22364+ }
22365+ decode -= 4;
22366+ #endif
22367+
22368+ // do the remnants
22369+ #if stbir__coder_min_num < 4
22370+ STBIR_NO_UNROLL_LOOP_START
22371+ while( decode < decode_end )
22372+ {
22373+ STBIR_NO_UNROLL(decode);
22374+ decode[0] = stbir__half_to_float(input[stbir__decode_order0]);
22375+ #if stbir__coder_min_num >= 2
22376+ decode[1] = stbir__half_to_float(input[stbir__decode_order1]);
22377+ #endif
22378+ #if stbir__coder_min_num >= 3
22379+ decode[2] = stbir__half_to_float(input[stbir__decode_order2]);
22380+ #endif
22381+ decode += stbir__coder_min_num;
22382+ input += stbir__coder_min_num;
22383+ }
22384+ #endif
22385+ return decode_end;
22386+}
22387+
22388+static void STBIR__CODER_NAME( stbir__encode_half_float_linear )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
22389+{
22390+ stbir__FP16 STBIR_SIMD_STREAMOUT_PTR( * ) output = (stbir__FP16*) outputp;
22391+ stbir__FP16 * end_output = ( (stbir__FP16*) output ) + width_times_channels;
22392+
22393+ #ifdef STBIR_SIMD
22394+ // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
22395+ // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
22396+ STBIR_SIMD_NO_UNROLL_LOOP_START
22397+ while ( ( end_output - output ) >= 8 )
22398+ {
22399+ STBIR_SIMD_NO_UNROLL(encode);
22400+ #ifdef stbir__decode_swizzle
22401+ #ifdef STBIR_SIMD8
22402+ {
22403+ stbir__simdf8 of;
22404+ stbir__simdf8_load( of, encode );
22405+ stbir__encode_simdf8_unflip( of );
22406+ stbir__float_to_half_SIMD( output, (float*)&of );
22407+ }
22408+ #else
22409+ {
22410+ stbir__simdf of[2];
22411+ stbir__simdf_load( of[0], encode );
22412+ stbir__simdf_load( of[1], encode+4 );
22413+ stbir__encode_simdf4_unflip( of[0] );
22414+ stbir__encode_simdf4_unflip( of[1] );
22415+ stbir__float_to_half_SIMD( output, (float*)of );
22416+ }
22417+ #endif
22418+ #else
22419+ stbir__float_to_half_SIMD( output, encode );
22420+ #endif
22421+ encode += 8;
22422+ output += 8;
22423+ }
22424+ #endif
22425+
22426+ // try to do blocks of 4 when you can
22427+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
22428+ output += 4;
22429+ STBIR_SIMD_NO_UNROLL_LOOP_START
22430+ while( output <= end_output )
22431+ {
22432+ float e0, e1, e2, e3;
22433+ STBIR_SIMD_NO_UNROLL(output);
22434+ e0 = encode[stbir__encode_order0];
22435+ e1 = encode[stbir__encode_order1];
22436+ e2 = encode[stbir__encode_order2];
22437+ e3 = encode[stbir__encode_order3];
22438+ output[0-4] = stbir__float_to_half(e0);
22439+ output[1-4] = stbir__float_to_half(e1);
22440+ output[2-4] = stbir__float_to_half(e2);
22441+ output[3-4] = stbir__float_to_half(e3);
22442+ output += 4;
22443+ encode += 4;
22444+ }
22445+ output -= 4;
22446+ #endif
22447+
22448+ // do the remnants
22449+ #if stbir__coder_min_num < 4
22450+ STBIR_NO_UNROLL_LOOP_START
22451+ while( output < end_output )
22452+ {
22453+ float e0;
22454+ #if stbir__coder_min_num >= 2
22455+ float e1;
22456+ #endif
22457+ #if stbir__coder_min_num >= 3
22458+ float e2;
22459+ #endif
22460+ STBIR_NO_UNROLL(output);
22461+ e0 = encode[stbir__encode_order0];
22462+ #if stbir__coder_min_num >= 2
22463+ e1 = encode[stbir__encode_order1];
22464+ #endif
22465+ #if stbir__coder_min_num >= 3
22466+ e2 = encode[stbir__encode_order2];
22467+ #endif
22468+ output[0] = stbir__float_to_half(e0);
22469+ #if stbir__coder_min_num >= 2
22470+ output[1] = stbir__float_to_half(e1);
22471+ #endif
22472+ #if stbir__coder_min_num >= 3
22473+ output[2] = stbir__float_to_half(e2);
22474+ #endif
22475+ output += stbir__coder_min_num;
22476+ encode += stbir__coder_min_num;
22477+ }
22478+ #endif
22479+}
22480+
22481+static float * STBIR__CODER_NAME(stbir__decode_float_linear)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
22482+{
22483+ #ifdef stbir__decode_swizzle
22484+ float STBIR_STREAMOUT_PTR( * ) decode = decodep;
22485+ float * decode_end = (float*) decode + width_times_channels;
22486+ float const * input = (float const *)inputp;
22487+
22488+ #ifdef STBIR_SIMD
22489+ if ( width_times_channels >= 16 )
22490+ {
22491+ float const * end_input_m16 = input + width_times_channels - 16;
22492+ decode_end -= 16;
22493+ STBIR_NO_UNROLL_LOOP_START_INF_FOR
22494+ for(;;)
22495+ {
22496+ STBIR_NO_UNROLL(decode);
22497+ #ifdef STBIR_SIMD8
22498+ {
22499+ stbir__simdf8 of0,of1;
22500+ stbir__simdf8_load( of0, input );
22501+ stbir__simdf8_load( of1, input+8 );
22502+ stbir__decode_simdf8_flip( of0 );
22503+ stbir__decode_simdf8_flip( of1 );
22504+ stbir__simdf8_store( decode, of0 );
22505+ stbir__simdf8_store( decode+8, of1 );
22506+ }
22507+ #else
22508+ {
22509+ stbir__simdf of0,of1,of2,of3;
22510+ stbir__simdf_load( of0, input );
22511+ stbir__simdf_load( of1, input+4 );
22512+ stbir__simdf_load( of2, input+8 );
22513+ stbir__simdf_load( of3, input+12 );
22514+ stbir__decode_simdf4_flip( of0 );
22515+ stbir__decode_simdf4_flip( of1 );
22516+ stbir__decode_simdf4_flip( of2 );
22517+ stbir__decode_simdf4_flip( of3 );
22518+ stbir__simdf_store( decode, of0 );
22519+ stbir__simdf_store( decode+4, of1 );
22520+ stbir__simdf_store( decode+8, of2 );
22521+ stbir__simdf_store( decode+12, of3 );
22522+ }
22523+ #endif
22524+ decode += 16;
22525+ input += 16;
22526+ if ( decode <= decode_end )
22527+ continue;
22528+ if ( decode == ( decode_end + 16 ) )
22529+ break;
22530+ decode = decode_end; // backup and do last couple
22531+ input = end_input_m16;
22532+ }
22533+ return decode_end + 16;
22534+ }
22535+ #endif
22536+
22537+ // try to do blocks of 4 when you can
22538+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
22539+ decode += 4;
22540+ STBIR_SIMD_NO_UNROLL_LOOP_START
22541+ while( decode <= decode_end )
22542+ {
22543+ STBIR_SIMD_NO_UNROLL(decode);
22544+ decode[0-4] = input[stbir__decode_order0];
22545+ decode[1-4] = input[stbir__decode_order1];
22546+ decode[2-4] = input[stbir__decode_order2];
22547+ decode[3-4] = input[stbir__decode_order3];
22548+ decode += 4;
22549+ input += 4;
22550+ }
22551+ decode -= 4;
22552+ #endif
22553+
22554+ // do the remnants
22555+ #if stbir__coder_min_num < 4
22556+ STBIR_NO_UNROLL_LOOP_START
22557+ while( decode < decode_end )
22558+ {
22559+ STBIR_NO_UNROLL(decode);
22560+ decode[0] = input[stbir__decode_order0];
22561+ #if stbir__coder_min_num >= 2
22562+ decode[1] = input[stbir__decode_order1];
22563+ #endif
22564+ #if stbir__coder_min_num >= 3
22565+ decode[2] = input[stbir__decode_order2];
22566+ #endif
22567+ decode += stbir__coder_min_num;
22568+ input += stbir__coder_min_num;
22569+ }
22570+ #endif
22571+ return decode_end;
22572+
22573+ #else
22574+
22575+ if ( (void*)decodep != inputp )
22576+ STBIR_MEMCPY( decodep, inputp, width_times_channels * sizeof( float ) );
22577+
22578+ return decodep + width_times_channels;
22579+
22580+ #endif
22581+}
22582+
22583+static void STBIR__CODER_NAME( stbir__encode_float_linear )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
22584+{
22585+ #if !defined( STBIR_FLOAT_HIGH_CLAMP ) && !defined(STBIR_FLOAT_LOW_CLAMP) && !defined(stbir__decode_swizzle)
22586+
22587+ if ( (void*)outputp != (void*) encode )
22588+ STBIR_MEMCPY( outputp, encode, width_times_channels * sizeof( float ) );
22589+
22590+ #else
22591+
22592+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = (float*) outputp;
22593+ float * end_output = ( (float*) output ) + width_times_channels;
22594+
22595+ #ifdef STBIR_FLOAT_HIGH_CLAMP
22596+ #define stbir_scalar_hi_clamp( v ) if ( v > STBIR_FLOAT_HIGH_CLAMP ) v = STBIR_FLOAT_HIGH_CLAMP;
22597+ #else
22598+ #define stbir_scalar_hi_clamp( v )
22599+ #endif
22600+ #ifdef STBIR_FLOAT_LOW_CLAMP
22601+ #define stbir_scalar_lo_clamp( v ) if ( v < STBIR_FLOAT_LOW_CLAMP ) v = STBIR_FLOAT_LOW_CLAMP;
22602+ #else
22603+ #define stbir_scalar_lo_clamp( v )
22604+ #endif
22605+
22606+ #ifdef STBIR_SIMD
22607+
22608+ #ifdef STBIR_FLOAT_HIGH_CLAMP
22609+ const stbir__simdfX high_clamp = stbir__simdf_frepX(STBIR_FLOAT_HIGH_CLAMP);
22610+ #endif
22611+ #ifdef STBIR_FLOAT_LOW_CLAMP
22612+ const stbir__simdfX low_clamp = stbir__simdf_frepX(STBIR_FLOAT_LOW_CLAMP);
22613+ #endif
22614+
22615+ // do as many double width blocks as fit and let the four-at-a-time and scalar loops below finish the row. (with an
22616+ // output callback we encode in place, so we can't use the usual trick of backing up and redoing the last block)
22617+ STBIR_SIMD_NO_UNROLL_LOOP_START
22618+ while ( ( end_output - output ) >= ( stbir__simdfX_float_count * 2 ) )
22619+ {
22620+ stbir__simdfX e0, e1;
22621+ STBIR_SIMD_NO_UNROLL(encode);
22622+ stbir__simdfX_load( e0, encode );
22623+ stbir__simdfX_load( e1, encode+stbir__simdfX_float_count );
22624+#ifdef STBIR_FLOAT_HIGH_CLAMP
22625+ stbir__simdfX_min( e0, e0, high_clamp );
22626+ stbir__simdfX_min( e1, e1, high_clamp );
22627+#endif
22628+#ifdef STBIR_FLOAT_LOW_CLAMP
22629+ stbir__simdfX_max( e0, e0, low_clamp );
22630+ stbir__simdfX_max( e1, e1, low_clamp );
22631+#endif
22632+ stbir__encode_simdfX_unflip( e0 );
22633+ stbir__encode_simdfX_unflip( e1 );
22634+ stbir__simdfX_store( output, e0 );
22635+ stbir__simdfX_store( output+stbir__simdfX_float_count, e1 );
22636+ encode += stbir__simdfX_float_count * 2;
22637+ output += stbir__simdfX_float_count * 2;
22638+ }
22639+
22640+ // try to do blocks of 4 when you can
22641+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
22642+ output += 4;
22643+ STBIR_NO_UNROLL_LOOP_START
22644+ while( output <= end_output )
22645+ {
22646+ stbir__simdf e0;
22647+ STBIR_NO_UNROLL(encode);
22648+ stbir__simdf_load( e0, encode );
22649+#ifdef STBIR_FLOAT_HIGH_CLAMP
22650+ stbir__simdf_min( e0, e0, stbir__simdfX_cast_to_simdf4( high_clamp ) );
22651+#endif
22652+#ifdef STBIR_FLOAT_LOW_CLAMP
22653+ stbir__simdf_max( e0, e0, stbir__simdfX_cast_to_simdf4( low_clamp ) );
22654+#endif
22655+ stbir__encode_simdf4_unflip( e0 );
22656+ stbir__simdf_store( output-4, e0 );
22657+ output += 4;
22658+ encode += 4;
22659+ }
22660+ output -= 4;
22661+ #endif
22662+
22663+ #else
22664+
22665+ // try to do blocks of 4 when you can
22666+ #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
22667+ output += 4;
22668+ STBIR_SIMD_NO_UNROLL_LOOP_START
22669+ while( output <= end_output )
22670+ {
22671+ // read everything before writing anything - with an output callback we encode in place, so this is an in-place unflip
22672+ float e0 = encode[ stbir__encode_order0 ];
22673+ float e1 = encode[ stbir__encode_order1 ];
22674+ float e2 = encode[ stbir__encode_order2 ];
22675+ float e3 = encode[ stbir__encode_order3 ];
22676+ STBIR_SIMD_NO_UNROLL(encode);
22677+ stbir_scalar_hi_clamp( e0 );
22678+ stbir_scalar_lo_clamp( e0 );
22679+ stbir_scalar_hi_clamp( e1 );
22680+ stbir_scalar_lo_clamp( e1 );
22681+ stbir_scalar_hi_clamp( e2 );
22682+ stbir_scalar_lo_clamp( e2 );
22683+ stbir_scalar_hi_clamp( e3 );
22684+ stbir_scalar_lo_clamp( e3 );
22685+ output[0-4] = e0;
22686+ output[1-4] = e1;
22687+ output[2-4] = e2;
22688+ output[3-4] = e3;
22689+ output += 4;
22690+ encode += 4;
22691+ }
22692+ output -= 4;
22693+
22694+ #endif
22695+
22696+ #endif
22697+
22698+ // do the remnants
22699+ #if stbir__coder_min_num < 4
22700+ STBIR_NO_UNROLL_LOOP_START
22701+ while( output < end_output )
22702+ {
22703+ float e0 = encode[ stbir__encode_order0 ];
22704+ #if stbir__coder_min_num >= 2
22705+ float e1 = encode[ stbir__encode_order1 ];
22706+ #endif
22707+ #if stbir__coder_min_num >= 3
22708+ float e2 = encode[ stbir__encode_order2 ];
22709+ #endif
22710+ STBIR_NO_UNROLL(encode);
22711+ stbir_scalar_hi_clamp( e0 ); stbir_scalar_lo_clamp( e0 ); output[0] = e0;
22712+ #if stbir__coder_min_num >= 2
22713+ stbir_scalar_hi_clamp( e1 ); stbir_scalar_lo_clamp( e1 ); output[1] = e1;
22714+ #endif
22715+ #if stbir__coder_min_num >= 3
22716+ stbir_scalar_hi_clamp( e2 ); stbir_scalar_lo_clamp( e2 ); output[2] = e2;
22717+ #endif
22718+ output += stbir__coder_min_num;
22719+ encode += stbir__coder_min_num;
22720+ }
22721+ #endif
22722+
22723+ #endif
22724+}
22725+
22726+#undef stbir__decode_suffix
22727+#undef stbir__decode_simdf8_flip
22728+#undef stbir__decode_simdf8_flip_i
22729+#undef stbir__decode_simdf4_flip_i
22730+#undef stbir__decode_simdi_flip_u8
22731+#undef stbir__decode_simdi_flip_u16
22732+#ifdef STBIR__SWZ8
22733+#undef STBIR__SWZ8
22734+#undef STBIR__SWZ16
22735+#endif
22736+#undef stbir__decode_simdf4_flip
22737+#undef stbir__decode_order0
22738+#undef stbir__decode_order1
22739+#undef stbir__decode_order2
22740+#undef stbir__decode_order3
22741+#undef stbir__encode_order0
22742+#undef stbir__encode_order1
22743+#undef stbir__encode_order2
22744+#undef stbir__encode_order3
22745+#undef stbir__encode_simdf8_unflip
22746+#undef stbir__encode_simdf16_unflip
22747+#undef stbir__decode_simdf16_flip_i
22748+#undef stbir__decode_simdi8_flip_u16
22749+#undef stbir__encode_simdf4_unflip
22750+#undef stbir__encode_simdfX_unflip
22751+#undef STBIR__CODER_NAME
22752+#undef stbir__coder_min_num
22753+#undef stbir__decode_swizzle
22754+#undef stbir_scalar_hi_clamp
22755+#undef stbir_scalar_lo_clamp
22756+#undef STB_IMAGE_RESIZE_DO_CODERS
22757+
22758+#elif defined( STB_IMAGE_RESIZE_DO_VERTICALS)
22759+
22760+#ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
22761+#define STBIR_chans( start, end ) STBIR_strs_join14(start,STBIR__vertical_channels,end,_cont)
22762+#else
22763+#define STBIR_chans( start, end ) STBIR_strs_join1(start,STBIR__vertical_channels,end)
22764+#endif
22765+
22766+#if STBIR__vertical_channels >= 1
22767+#define stbIF0( code ) code
22768+#else
22769+#define stbIF0( code )
22770+#endif
22771+#if STBIR__vertical_channels >= 2
22772+#define stbIF1( code ) code
22773+#else
22774+#define stbIF1( code )
22775+#endif
22776+#if STBIR__vertical_channels >= 3
22777+#define stbIF2( code ) code
22778+#else
22779+#define stbIF2( code )
22780+#endif
22781+#if STBIR__vertical_channels >= 4
22782+#define stbIF3( code ) code
22783+#else
22784+#define stbIF3( code )
22785+#endif
22786+#if STBIR__vertical_channels >= 5
22787+#define stbIF4( code ) code
22788+#else
22789+#define stbIF4( code )
22790+#endif
22791+#if STBIR__vertical_channels >= 6
22792+#define stbIF5( code ) code
22793+#else
22794+#define stbIF5( code )
22795+#endif
22796+#if STBIR__vertical_channels >= 7
22797+#define stbIF6( code ) code
22798+#else
22799+#define stbIF6( code )
22800+#endif
22801+#if STBIR__vertical_channels >= 8
22802+#define stbIF7( code ) code
22803+#else
22804+#define stbIF7( code )
22805+#endif
22806+
22807+static void STBIR_chans( stbir__vertical_scatter_with_,_coeffs)( float ** outputs, float const * vertical_coefficients, float const * input, float const * input_end )
22808+{
22809+ stbIF0( float STBIR_SIMD_STREAMOUT_PTR( * ) output0 = outputs[0]; float c0s = vertical_coefficients[0]; )
22810+ stbIF1( float STBIR_SIMD_STREAMOUT_PTR( * ) output1 = outputs[1]; float c1s = vertical_coefficients[1]; )
22811+ stbIF2( float STBIR_SIMD_STREAMOUT_PTR( * ) output2 = outputs[2]; float c2s = vertical_coefficients[2]; )
22812+ stbIF3( float STBIR_SIMD_STREAMOUT_PTR( * ) output3 = outputs[3]; float c3s = vertical_coefficients[3]; )
22813+ stbIF4( float STBIR_SIMD_STREAMOUT_PTR( * ) output4 = outputs[4]; float c4s = vertical_coefficients[4]; )
22814+ stbIF5( float STBIR_SIMD_STREAMOUT_PTR( * ) output5 = outputs[5]; float c5s = vertical_coefficients[5]; )
22815+ stbIF6( float STBIR_SIMD_STREAMOUT_PTR( * ) output6 = outputs[6]; float c6s = vertical_coefficients[6]; )
22816+ stbIF7( float STBIR_SIMD_STREAMOUT_PTR( * ) output7 = outputs[7]; float c7s = vertical_coefficients[7]; )
22817+
22818+ #ifdef STBIR_SIMD
22819+ {
22820+ stbIF0(stbir__simdfX c0 = stbir__simdf_frepX( c0s ); )
22821+ stbIF1(stbir__simdfX c1 = stbir__simdf_frepX( c1s ); )
22822+ stbIF2(stbir__simdfX c2 = stbir__simdf_frepX( c2s ); )
22823+ stbIF3(stbir__simdfX c3 = stbir__simdf_frepX( c3s ); )
22824+ stbIF4(stbir__simdfX c4 = stbir__simdf_frepX( c4s ); )
22825+ stbIF5(stbir__simdfX c5 = stbir__simdf_frepX( c5s ); )
22826+ stbIF6(stbir__simdfX c6 = stbir__simdf_frepX( c6s ); )
22827+ stbIF7(stbir__simdfX c7 = stbir__simdf_frepX( c7s ); )
22828+ STBIR_SIMD_NO_UNROLL_LOOP_START
22829+ while ( ( (char*)input_end - (char*) input ) >= (16*stbir__simdfX_float_count) )
22830+ {
22831+ stbir__simdfX o0, o1, o2, o3, r0, r1, r2, r3;
22832+ STBIR_SIMD_NO_UNROLL(output0);
22833+
22834+ stbir__simdfX_load( r0, input ); stbir__simdfX_load( r1, input+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input+(3*stbir__simdfX_float_count) );
22835+
22836+ #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
22837+ stbIF0( stbir__simdfX_load( o0, output0 ); stbir__simdfX_load( o1, output0+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output0+(3*stbir__simdfX_float_count) );
22838+ stbir__simdfX_madd( o0, o0, r0, c0 ); stbir__simdfX_madd( o1, o1, r1, c0 ); stbir__simdfX_madd( o2, o2, r2, c0 ); stbir__simdfX_madd( o3, o3, r3, c0 );
22839+ stbir__simdfX_store( output0, o0 ); stbir__simdfX_store( output0+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output0+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output0+(3*stbir__simdfX_float_count), o3 ); )
22840+ stbIF1( stbir__simdfX_load( o0, output1 ); stbir__simdfX_load( o1, output1+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output1+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output1+(3*stbir__simdfX_float_count) );
22841+ stbir__simdfX_madd( o0, o0, r0, c1 ); stbir__simdfX_madd( o1, o1, r1, c1 ); stbir__simdfX_madd( o2, o2, r2, c1 ); stbir__simdfX_madd( o3, o3, r3, c1 );
22842+ stbir__simdfX_store( output1, o0 ); stbir__simdfX_store( output1+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output1+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output1+(3*stbir__simdfX_float_count), o3 ); )
22843+ stbIF2( stbir__simdfX_load( o0, output2 ); stbir__simdfX_load( o1, output2+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output2+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output2+(3*stbir__simdfX_float_count) );
22844+ stbir__simdfX_madd( o0, o0, r0, c2 ); stbir__simdfX_madd( o1, o1, r1, c2 ); stbir__simdfX_madd( o2, o2, r2, c2 ); stbir__simdfX_madd( o3, o3, r3, c2 );
22845+ stbir__simdfX_store( output2, o0 ); stbir__simdfX_store( output2+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output2+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output2+(3*stbir__simdfX_float_count), o3 ); )
22846+ stbIF3( stbir__simdfX_load( o0, output3 ); stbir__simdfX_load( o1, output3+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output3+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output3+(3*stbir__simdfX_float_count) );
22847+ stbir__simdfX_madd( o0, o0, r0, c3 ); stbir__simdfX_madd( o1, o1, r1, c3 ); stbir__simdfX_madd( o2, o2, r2, c3 ); stbir__simdfX_madd( o3, o3, r3, c3 );
22848+ stbir__simdfX_store( output3, o0 ); stbir__simdfX_store( output3+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output3+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output3+(3*stbir__simdfX_float_count), o3 ); )
22849+ stbIF4( stbir__simdfX_load( o0, output4 ); stbir__simdfX_load( o1, output4+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output4+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output4+(3*stbir__simdfX_float_count) );
22850+ stbir__simdfX_madd( o0, o0, r0, c4 ); stbir__simdfX_madd( o1, o1, r1, c4 ); stbir__simdfX_madd( o2, o2, r2, c4 ); stbir__simdfX_madd( o3, o3, r3, c4 );
22851+ stbir__simdfX_store( output4, o0 ); stbir__simdfX_store( output4+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output4+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output4+(3*stbir__simdfX_float_count), o3 ); )
22852+ stbIF5( stbir__simdfX_load( o0, output5 ); stbir__simdfX_load( o1, output5+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output5+(2*stbir__simdfX_float_count)); stbir__simdfX_load( o3, output5+(3*stbir__simdfX_float_count) );
22853+ stbir__simdfX_madd( o0, o0, r0, c5 ); stbir__simdfX_madd( o1, o1, r1, c5 ); stbir__simdfX_madd( o2, o2, r2, c5 ); stbir__simdfX_madd( o3, o3, r3, c5 );
22854+ stbir__simdfX_store( output5, o0 ); stbir__simdfX_store( output5+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output5+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output5+(3*stbir__simdfX_float_count), o3 ); )
22855+ stbIF6( stbir__simdfX_load( o0, output6 ); stbir__simdfX_load( o1, output6+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output6+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output6+(3*stbir__simdfX_float_count) );
22856+ stbir__simdfX_madd( o0, o0, r0, c6 ); stbir__simdfX_madd( o1, o1, r1, c6 ); stbir__simdfX_madd( o2, o2, r2, c6 ); stbir__simdfX_madd( o3, o3, r3, c6 );
22857+ stbir__simdfX_store( output6, o0 ); stbir__simdfX_store( output6+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output6+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output6+(3*stbir__simdfX_float_count), o3 ); )
22858+ stbIF7( stbir__simdfX_load( o0, output7 ); stbir__simdfX_load( o1, output7+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output7+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output7+(3*stbir__simdfX_float_count) );
22859+ stbir__simdfX_madd( o0, o0, r0, c7 ); stbir__simdfX_madd( o1, o1, r1, c7 ); stbir__simdfX_madd( o2, o2, r2, c7 ); stbir__simdfX_madd( o3, o3, r3, c7 );
22860+ stbir__simdfX_store( output7, o0 ); stbir__simdfX_store( output7+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output7+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output7+(3*stbir__simdfX_float_count), o3 ); )
22861+ #else
22862+ stbIF0( stbir__simdfX_mult( o0, r0, c0 ); stbir__simdfX_mult( o1, r1, c0 ); stbir__simdfX_mult( o2, r2, c0 ); stbir__simdfX_mult( o3, r3, c0 );
22863+ stbir__simdfX_store( output0, o0 ); stbir__simdfX_store( output0+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output0+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output0+(3*stbir__simdfX_float_count), o3 ); )
22864+ stbIF1( stbir__simdfX_mult( o0, r0, c1 ); stbir__simdfX_mult( o1, r1, c1 ); stbir__simdfX_mult( o2, r2, c1 ); stbir__simdfX_mult( o3, r3, c1 );
22865+ stbir__simdfX_store( output1, o0 ); stbir__simdfX_store( output1+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output1+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output1+(3*stbir__simdfX_float_count), o3 ); )
22866+ stbIF2( stbir__simdfX_mult( o0, r0, c2 ); stbir__simdfX_mult( o1, r1, c2 ); stbir__simdfX_mult( o2, r2, c2 ); stbir__simdfX_mult( o3, r3, c2 );
22867+ stbir__simdfX_store( output2, o0 ); stbir__simdfX_store( output2+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output2+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output2+(3*stbir__simdfX_float_count), o3 ); )
22868+ stbIF3( stbir__simdfX_mult( o0, r0, c3 ); stbir__simdfX_mult( o1, r1, c3 ); stbir__simdfX_mult( o2, r2, c3 ); stbir__simdfX_mult( o3, r3, c3 );
22869+ stbir__simdfX_store( output3, o0 ); stbir__simdfX_store( output3+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output3+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output3+(3*stbir__simdfX_float_count), o3 ); )
22870+ stbIF4( stbir__simdfX_mult( o0, r0, c4 ); stbir__simdfX_mult( o1, r1, c4 ); stbir__simdfX_mult( o2, r2, c4 ); stbir__simdfX_mult( o3, r3, c4 );
22871+ stbir__simdfX_store( output4, o0 ); stbir__simdfX_store( output4+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output4+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output4+(3*stbir__simdfX_float_count), o3 ); )
22872+ stbIF5( stbir__simdfX_mult( o0, r0, c5 ); stbir__simdfX_mult( o1, r1, c5 ); stbir__simdfX_mult( o2, r2, c5 ); stbir__simdfX_mult( o3, r3, c5 );
22873+ stbir__simdfX_store( output5, o0 ); stbir__simdfX_store( output5+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output5+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output5+(3*stbir__simdfX_float_count), o3 ); )
22874+ stbIF6( stbir__simdfX_mult( o0, r0, c6 ); stbir__simdfX_mult( o1, r1, c6 ); stbir__simdfX_mult( o2, r2, c6 ); stbir__simdfX_mult( o3, r3, c6 );
22875+ stbir__simdfX_store( output6, o0 ); stbir__simdfX_store( output6+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output6+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output6+(3*stbir__simdfX_float_count), o3 ); )
22876+ stbIF7( stbir__simdfX_mult( o0, r0, c7 ); stbir__simdfX_mult( o1, r1, c7 ); stbir__simdfX_mult( o2, r2, c7 ); stbir__simdfX_mult( o3, r3, c7 );
22877+ stbir__simdfX_store( output7, o0 ); stbir__simdfX_store( output7+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output7+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output7+(3*stbir__simdfX_float_count), o3 ); )
22878+ #endif
22879+
22880+ input += (4*stbir__simdfX_float_count);
22881+ stbIF0( output0 += (4*stbir__simdfX_float_count); )
22882+ stbIF1( output1 += (4*stbir__simdfX_float_count); )
22883+ stbIF2( output2 += (4*stbir__simdfX_float_count); )
22884+ stbIF3( output3 += (4*stbir__simdfX_float_count); )
22885+ stbIF4( output4 += (4*stbir__simdfX_float_count); )
22886+ stbIF5( output5 += (4*stbir__simdfX_float_count); )
22887+ stbIF6( output6 += (4*stbir__simdfX_float_count); )
22888+ stbIF7( output7 += (4*stbir__simdfX_float_count); )
22889+ }
22890+ STBIR_SIMD_NO_UNROLL_LOOP_START
22891+ while ( ( (char*)input_end - (char*) input ) >= 16 )
22892+ {
22893+ stbir__simdf o0, r0;
22894+ STBIR_SIMD_NO_UNROLL(output0);
22895+
22896+ stbir__simdf_load( r0, input );
22897+
22898+ #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
22899+ stbIF0( stbir__simdf_load( o0, output0 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c0 ) ); stbir__simdf_store( output0, o0 ); )
22900+ stbIF1( stbir__simdf_load( o0, output1 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c1 ) ); stbir__simdf_store( output1, o0 ); )
22901+ stbIF2( stbir__simdf_load( o0, output2 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c2 ) ); stbir__simdf_store( output2, o0 ); )
22902+ stbIF3( stbir__simdf_load( o0, output3 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c3 ) ); stbir__simdf_store( output3, o0 ); )
22903+ stbIF4( stbir__simdf_load( o0, output4 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c4 ) ); stbir__simdf_store( output4, o0 ); )
22904+ stbIF5( stbir__simdf_load( o0, output5 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c5 ) ); stbir__simdf_store( output5, o0 ); )
22905+ stbIF6( stbir__simdf_load( o0, output6 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c6 ) ); stbir__simdf_store( output6, o0 ); )
22906+ stbIF7( stbir__simdf_load( o0, output7 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c7 ) ); stbir__simdf_store( output7, o0 ); )
22907+ #else
22908+ stbIF0( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c0 ) ); stbir__simdf_store( output0, o0 ); )
22909+ stbIF1( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c1 ) ); stbir__simdf_store( output1, o0 ); )
22910+ stbIF2( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c2 ) ); stbir__simdf_store( output2, o0 ); )
22911+ stbIF3( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c3 ) ); stbir__simdf_store( output3, o0 ); )
22912+ stbIF4( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c4 ) ); stbir__simdf_store( output4, o0 ); )
22913+ stbIF5( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c5 ) ); stbir__simdf_store( output5, o0 ); )
22914+ stbIF6( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c6 ) ); stbir__simdf_store( output6, o0 ); )
22915+ stbIF7( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c7 ) ); stbir__simdf_store( output7, o0 ); )
22916+ #endif
22917+
22918+ input += 4;
22919+ stbIF0( output0 += 4; )
22920+ stbIF1( output1 += 4; )
22921+ stbIF2( output2 += 4; )
22922+ stbIF3( output3 += 4; )
22923+ stbIF4( output4 += 4; )
22924+ stbIF5( output5 += 4; )
22925+ stbIF6( output6 += 4; )
22926+ stbIF7( output7 += 4; )
22927+ }
22928+ }
22929+ #else
22930+ STBIR_NO_UNROLL_LOOP_START
22931+ while ( ( (char*)input_end - (char*) input ) >= 16 )
22932+ {
22933+ float r0, r1, r2, r3;
22934+ STBIR_NO_UNROLL(input);
22935+
22936+ r0 = input[0], r1 = input[1], r2 = input[2], r3 = input[3];
22937+
22938+ #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
22939+ stbIF0( output0[0] += ( r0 * c0s ); output0[1] += ( r1 * c0s ); output0[2] += ( r2 * c0s ); output0[3] += ( r3 * c0s ); )
22940+ stbIF1( output1[0] += ( r0 * c1s ); output1[1] += ( r1 * c1s ); output1[2] += ( r2 * c1s ); output1[3] += ( r3 * c1s ); )
22941+ stbIF2( output2[0] += ( r0 * c2s ); output2[1] += ( r1 * c2s ); output2[2] += ( r2 * c2s ); output2[3] += ( r3 * c2s ); )
22942+ stbIF3( output3[0] += ( r0 * c3s ); output3[1] += ( r1 * c3s ); output3[2] += ( r2 * c3s ); output3[3] += ( r3 * c3s ); )
22943+ stbIF4( output4[0] += ( r0 * c4s ); output4[1] += ( r1 * c4s ); output4[2] += ( r2 * c4s ); output4[3] += ( r3 * c4s ); )
22944+ stbIF5( output5[0] += ( r0 * c5s ); output5[1] += ( r1 * c5s ); output5[2] += ( r2 * c5s ); output5[3] += ( r3 * c5s ); )
22945+ stbIF6( output6[0] += ( r0 * c6s ); output6[1] += ( r1 * c6s ); output6[2] += ( r2 * c6s ); output6[3] += ( r3 * c6s ); )
22946+ stbIF7( output7[0] += ( r0 * c7s ); output7[1] += ( r1 * c7s ); output7[2] += ( r2 * c7s ); output7[3] += ( r3 * c7s ); )
22947+ #else
22948+ stbIF0( output0[0] = ( r0 * c0s ); output0[1] = ( r1 * c0s ); output0[2] = ( r2 * c0s ); output0[3] = ( r3 * c0s ); )
22949+ stbIF1( output1[0] = ( r0 * c1s ); output1[1] = ( r1 * c1s ); output1[2] = ( r2 * c1s ); output1[3] = ( r3 * c1s ); )
22950+ stbIF2( output2[0] = ( r0 * c2s ); output2[1] = ( r1 * c2s ); output2[2] = ( r2 * c2s ); output2[3] = ( r3 * c2s ); )
22951+ stbIF3( output3[0] = ( r0 * c3s ); output3[1] = ( r1 * c3s ); output3[2] = ( r2 * c3s ); output3[3] = ( r3 * c3s ); )
22952+ stbIF4( output4[0] = ( r0 * c4s ); output4[1] = ( r1 * c4s ); output4[2] = ( r2 * c4s ); output4[3] = ( r3 * c4s ); )
22953+ stbIF5( output5[0] = ( r0 * c5s ); output5[1] = ( r1 * c5s ); output5[2] = ( r2 * c5s ); output5[3] = ( r3 * c5s ); )
22954+ stbIF6( output6[0] = ( r0 * c6s ); output6[1] = ( r1 * c6s ); output6[2] = ( r2 * c6s ); output6[3] = ( r3 * c6s ); )
22955+ stbIF7( output7[0] = ( r0 * c7s ); output7[1] = ( r1 * c7s ); output7[2] = ( r2 * c7s ); output7[3] = ( r3 * c7s ); )
22956+ #endif
22957+
22958+ input += 4;
22959+ stbIF0( output0 += 4; )
22960+ stbIF1( output1 += 4; )
22961+ stbIF2( output2 += 4; )
22962+ stbIF3( output3 += 4; )
22963+ stbIF4( output4 += 4; )
22964+ stbIF5( output5 += 4; )
22965+ stbIF6( output6 += 4; )
22966+ stbIF7( output7 += 4; )
22967+ }
22968+ #endif
22969+ STBIR_NO_UNROLL_LOOP_START
22970+ while ( input < input_end )
22971+ {
22972+ float r = input[0];
22973+ STBIR_NO_UNROLL(output0);
22974+
22975+ #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
22976+ stbIF0( output0[0] += ( r * c0s ); )
22977+ stbIF1( output1[0] += ( r * c1s ); )
22978+ stbIF2( output2[0] += ( r * c2s ); )
22979+ stbIF3( output3[0] += ( r * c3s ); )
22980+ stbIF4( output4[0] += ( r * c4s ); )
22981+ stbIF5( output5[0] += ( r * c5s ); )
22982+ stbIF6( output6[0] += ( r * c6s ); )
22983+ stbIF7( output7[0] += ( r * c7s ); )
22984+ #else
22985+ stbIF0( output0[0] = ( r * c0s ); )
22986+ stbIF1( output1[0] = ( r * c1s ); )
22987+ stbIF2( output2[0] = ( r * c2s ); )
22988+ stbIF3( output3[0] = ( r * c3s ); )
22989+ stbIF4( output4[0] = ( r * c4s ); )
22990+ stbIF5( output5[0] = ( r * c5s ); )
22991+ stbIF6( output6[0] = ( r * c6s ); )
22992+ stbIF7( output7[0] = ( r * c7s ); )
22993+ #endif
22994+
22995+ ++input;
22996+ stbIF0( ++output0; )
22997+ stbIF1( ++output1; )
22998+ stbIF2( ++output2; )
22999+ stbIF3( ++output3; )
23000+ stbIF4( ++output4; )
23001+ stbIF5( ++output5; )
23002+ stbIF6( ++output6; )
23003+ stbIF7( ++output7; )
23004+ }
23005+}
23006+
23007+static void STBIR_chans( stbir__vertical_gather_with_,_coeffs)( float * outputp, float const * vertical_coefficients, float const ** inputs, float const * input0_end )
23008+{
23009+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = outputp;
23010+
23011+ stbIF0( float const * input0 = inputs[0]; float c0s = vertical_coefficients[0]; )
23012+ stbIF1( float const * input1 = inputs[1]; float c1s = vertical_coefficients[1]; )
23013+ stbIF2( float const * input2 = inputs[2]; float c2s = vertical_coefficients[2]; )
23014+ stbIF3( float const * input3 = inputs[3]; float c3s = vertical_coefficients[3]; )
23015+ stbIF4( float const * input4 = inputs[4]; float c4s = vertical_coefficients[4]; )
23016+ stbIF5( float const * input5 = inputs[5]; float c5s = vertical_coefficients[5]; )
23017+ stbIF6( float const * input6 = inputs[6]; float c6s = vertical_coefficients[6]; )
23018+ stbIF7( float const * input7 = inputs[7]; float c7s = vertical_coefficients[7]; )
23019+
23020+#if ( STBIR__vertical_channels == 1 ) && !defined(STB_IMAGE_RESIZE_VERTICAL_CONTINUE)
23021+ // one coefficient within a millionth of 1 (any channel count, the single coefficient instantiation): a copy
23022+ if ( ( c0s >= (1.0f-0.000001f) ) && ( c0s <= (1.0f+0.000001f) ) )
23023+ {
23024+ STBIR_MEMCPY( output, input0, (char*)input0_end - (char*)input0 );
23025+ return;
23026+ }
23027+#endif
23028+
23029+ #ifdef STBIR_SIMD
23030+ {
23031+ stbIF0(stbir__simdfX c0 = stbir__simdf_frepX( c0s ); )
23032+ stbIF1(stbir__simdfX c1 = stbir__simdf_frepX( c1s ); )
23033+ stbIF2(stbir__simdfX c2 = stbir__simdf_frepX( c2s ); )
23034+ stbIF3(stbir__simdfX c3 = stbir__simdf_frepX( c3s ); )
23035+ stbIF4(stbir__simdfX c4 = stbir__simdf_frepX( c4s ); )
23036+ stbIF5(stbir__simdfX c5 = stbir__simdf_frepX( c5s ); )
23037+ stbIF6(stbir__simdfX c6 = stbir__simdf_frepX( c6s ); )
23038+ stbIF7(stbir__simdfX c7 = stbir__simdf_frepX( c7s ); )
23039+
23040+ STBIR_SIMD_NO_UNROLL_LOOP_START
23041+ while ( ( (char*)input0_end - (char*) input0 ) >= (16*stbir__simdfX_float_count) )
23042+ {
23043+ stbir__simdfX o0, o1, o2, o3, r0, r1, r2, r3;
23044+ STBIR_SIMD_NO_UNROLL(output);
23045+
23046+ // prefetch four loop iterations ahead (doesn't affect much for small resizes, but helps with big ones)
23047+ stbIF0( stbir__prefetch( input0 + (16*stbir__simdfX_float_count) ); )
23048+ stbIF1( stbir__prefetch( input1 + (16*stbir__simdfX_float_count) ); )
23049+ stbIF2( stbir__prefetch( input2 + (16*stbir__simdfX_float_count) ); )
23050+ stbIF3( stbir__prefetch( input3 + (16*stbir__simdfX_float_count) ); )
23051+ stbIF4( stbir__prefetch( input4 + (16*stbir__simdfX_float_count) ); )
23052+ stbIF5( stbir__prefetch( input5 + (16*stbir__simdfX_float_count) ); )
23053+ stbIF6( stbir__prefetch( input6 + (16*stbir__simdfX_float_count) ); )
23054+ stbIF7( stbir__prefetch( input7 + (16*stbir__simdfX_float_count) ); )
23055+
23056+ #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
23057+ stbIF0( stbir__simdfX_load( o0, output ); stbir__simdfX_load( o1, output+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output+(3*stbir__simdfX_float_count) );
23058+ stbir__simdfX_load( r0, input0 ); stbir__simdfX_load( r1, input0+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input0+(3*stbir__simdfX_float_count) );
23059+ stbir__simdfX_madd( o0, o0, r0, c0 ); stbir__simdfX_madd( o1, o1, r1, c0 ); stbir__simdfX_madd( o2, o2, r2, c0 ); stbir__simdfX_madd( o3, o3, r3, c0 ); )
23060+ #else
23061+ stbIF0( stbir__simdfX_load( r0, input0 ); stbir__simdfX_load( r1, input0+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input0+(3*stbir__simdfX_float_count) );
23062+ stbir__simdfX_mult( o0, r0, c0 ); stbir__simdfX_mult( o1, r1, c0 ); stbir__simdfX_mult( o2, r2, c0 ); stbir__simdfX_mult( o3, r3, c0 ); )
23063+ #endif
23064+
23065+ stbIF1( stbir__simdfX_load( r0, input1 ); stbir__simdfX_load( r1, input1+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input1+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input1+(3*stbir__simdfX_float_count) );
23066+ stbir__simdfX_madd( o0, o0, r0, c1 ); stbir__simdfX_madd( o1, o1, r1, c1 ); stbir__simdfX_madd( o2, o2, r2, c1 ); stbir__simdfX_madd( o3, o3, r3, c1 ); )
23067+ stbIF2( stbir__simdfX_load( r0, input2 ); stbir__simdfX_load( r1, input2+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input2+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input2+(3*stbir__simdfX_float_count) );
23068+ stbir__simdfX_madd( o0, o0, r0, c2 ); stbir__simdfX_madd( o1, o1, r1, c2 ); stbir__simdfX_madd( o2, o2, r2, c2 ); stbir__simdfX_madd( o3, o3, r3, c2 ); )
23069+ stbIF3( stbir__simdfX_load( r0, input3 ); stbir__simdfX_load( r1, input3+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input3+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input3+(3*stbir__simdfX_float_count) );
23070+ stbir__simdfX_madd( o0, o0, r0, c3 ); stbir__simdfX_madd( o1, o1, r1, c3 ); stbir__simdfX_madd( o2, o2, r2, c3 ); stbir__simdfX_madd( o3, o3, r3, c3 ); )
23071+ stbIF4( stbir__simdfX_load( r0, input4 ); stbir__simdfX_load( r1, input4+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input4+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input4+(3*stbir__simdfX_float_count) );
23072+ stbir__simdfX_madd( o0, o0, r0, c4 ); stbir__simdfX_madd( o1, o1, r1, c4 ); stbir__simdfX_madd( o2, o2, r2, c4 ); stbir__simdfX_madd( o3, o3, r3, c4 ); )
23073+ stbIF5( stbir__simdfX_load( r0, input5 ); stbir__simdfX_load( r1, input5+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input5+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input5+(3*stbir__simdfX_float_count) );
23074+ stbir__simdfX_madd( o0, o0, r0, c5 ); stbir__simdfX_madd( o1, o1, r1, c5 ); stbir__simdfX_madd( o2, o2, r2, c5 ); stbir__simdfX_madd( o3, o3, r3, c5 ); )
23075+ stbIF6( stbir__simdfX_load( r0, input6 ); stbir__simdfX_load( r1, input6+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input6+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input6+(3*stbir__simdfX_float_count) );
23076+ stbir__simdfX_madd( o0, o0, r0, c6 ); stbir__simdfX_madd( o1, o1, r1, c6 ); stbir__simdfX_madd( o2, o2, r2, c6 ); stbir__simdfX_madd( o3, o3, r3, c6 ); )
23077+ stbIF7( stbir__simdfX_load( r0, input7 ); stbir__simdfX_load( r1, input7+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input7+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input7+(3*stbir__simdfX_float_count) );
23078+ stbir__simdfX_madd( o0, o0, r0, c7 ); stbir__simdfX_madd( o1, o1, r1, c7 ); stbir__simdfX_madd( o2, o2, r2, c7 ); stbir__simdfX_madd( o3, o3, r3, c7 ); )
23079+
23080+ stbir__simdfX_store( output, o0 ); stbir__simdfX_store( output+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output+(3*stbir__simdfX_float_count), o3 );
23081+ output += (4*stbir__simdfX_float_count);
23082+ stbIF0( input0 += (4*stbir__simdfX_float_count); )
23083+ stbIF1( input1 += (4*stbir__simdfX_float_count); )
23084+ stbIF2( input2 += (4*stbir__simdfX_float_count); )
23085+ stbIF3( input3 += (4*stbir__simdfX_float_count); )
23086+ stbIF4( input4 += (4*stbir__simdfX_float_count); )
23087+ stbIF5( input5 += (4*stbir__simdfX_float_count); )
23088+ stbIF6( input6 += (4*stbir__simdfX_float_count); )
23089+ stbIF7( input7 += (4*stbir__simdfX_float_count); )
23090+ }
23091+
23092+ STBIR_SIMD_NO_UNROLL_LOOP_START
23093+ while ( ( (char*)input0_end - (char*) input0 ) >= 16 )
23094+ {
23095+ stbir__simdf o0, r0;
23096+ STBIR_SIMD_NO_UNROLL(output);
23097+
23098+ #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
23099+ stbIF0( stbir__simdf_load( o0, output ); stbir__simdf_load( r0, input0 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c0 ) ); )
23100+ #else
23101+ stbIF0( stbir__simdf_load( r0, input0 ); stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c0 ) ); )
23102+ #endif
23103+ stbIF1( stbir__simdf_load( r0, input1 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c1 ) ); )
23104+ stbIF2( stbir__simdf_load( r0, input2 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c2 ) ); )
23105+ stbIF3( stbir__simdf_load( r0, input3 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c3 ) ); )
23106+ stbIF4( stbir__simdf_load( r0, input4 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c4 ) ); )
23107+ stbIF5( stbir__simdf_load( r0, input5 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c5 ) ); )
23108+ stbIF6( stbir__simdf_load( r0, input6 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c6 ) ); )
23109+ stbIF7( stbir__simdf_load( r0, input7 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c7 ) ); )
23110+
23111+ stbir__simdf_store( output, o0 );
23112+ output += 4;
23113+ stbIF0( input0 += 4; )
23114+ stbIF1( input1 += 4; )
23115+ stbIF2( input2 += 4; )
23116+ stbIF3( input3 += 4; )
23117+ stbIF4( input4 += 4; )
23118+ stbIF5( input5 += 4; )
23119+ stbIF6( input6 += 4; )
23120+ stbIF7( input7 += 4; )
23121+ }
23122+ }
23123+ #else
23124+ STBIR_NO_UNROLL_LOOP_START
23125+ while ( ( (char*)input0_end - (char*) input0 ) >= 16 )
23126+ {
23127+ float o0, o1, o2, o3;
23128+ STBIR_NO_UNROLL(output);
23129+ #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
23130+ stbIF0( o0 = output[0] + input0[0] * c0s; o1 = output[1] + input0[1] * c0s; o2 = output[2] + input0[2] * c0s; o3 = output[3] + input0[3] * c0s; )
23131+ #else
23132+ stbIF0( o0 = input0[0] * c0s; o1 = input0[1] * c0s; o2 = input0[2] * c0s; o3 = input0[3] * c0s; )
23133+ #endif
23134+ stbIF1( o0 += input1[0] * c1s; o1 += input1[1] * c1s; o2 += input1[2] * c1s; o3 += input1[3] * c1s; )
23135+ stbIF2( o0 += input2[0] * c2s; o1 += input2[1] * c2s; o2 += input2[2] * c2s; o3 += input2[3] * c2s; )
23136+ stbIF3( o0 += input3[0] * c3s; o1 += input3[1] * c3s; o2 += input3[2] * c3s; o3 += input3[3] * c3s; )
23137+ stbIF4( o0 += input4[0] * c4s; o1 += input4[1] * c4s; o2 += input4[2] * c4s; o3 += input4[3] * c4s; )
23138+ stbIF5( o0 += input5[0] * c5s; o1 += input5[1] * c5s; o2 += input5[2] * c5s; o3 += input5[3] * c5s; )
23139+ stbIF6( o0 += input6[0] * c6s; o1 += input6[1] * c6s; o2 += input6[2] * c6s; o3 += input6[3] * c6s; )
23140+ stbIF7( o0 += input7[0] * c7s; o1 += input7[1] * c7s; o2 += input7[2] * c7s; o3 += input7[3] * c7s; )
23141+ output[0] = o0; output[1] = o1; output[2] = o2; output[3] = o3;
23142+ output += 4;
23143+ stbIF0( input0 += 4; )
23144+ stbIF1( input1 += 4; )
23145+ stbIF2( input2 += 4; )
23146+ stbIF3( input3 += 4; )
23147+ stbIF4( input4 += 4; )
23148+ stbIF5( input5 += 4; )
23149+ stbIF6( input6 += 4; )
23150+ stbIF7( input7 += 4; )
23151+ }
23152+ #endif
23153+ STBIR_NO_UNROLL_LOOP_START
23154+ while ( input0 < input0_end )
23155+ {
23156+ float o0;
23157+ STBIR_NO_UNROLL(output);
23158+ #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
23159+ stbIF0( o0 = output[0] + input0[0] * c0s; )
23160+ #else
23161+ stbIF0( o0 = input0[0] * c0s; )
23162+ #endif
23163+ stbIF1( o0 += input1[0] * c1s; )
23164+ stbIF2( o0 += input2[0] * c2s; )
23165+ stbIF3( o0 += input3[0] * c3s; )
23166+ stbIF4( o0 += input4[0] * c4s; )
23167+ stbIF5( o0 += input5[0] * c5s; )
23168+ stbIF6( o0 += input6[0] * c6s; )
23169+ stbIF7( o0 += input7[0] * c7s; )
23170+ output[0] = o0;
23171+ ++output;
23172+ stbIF0( ++input0; )
23173+ stbIF1( ++input1; )
23174+ stbIF2( ++input2; )
23175+ stbIF3( ++input3; )
23176+ stbIF4( ++input4; )
23177+ stbIF5( ++input5; )
23178+ stbIF6( ++input6; )
23179+ stbIF7( ++input7; )
23180+ }
23181+}
23182+
23183+#undef stbIF0
23184+#undef stbIF1
23185+#undef stbIF2
23186+#undef stbIF3
23187+#undef stbIF4
23188+#undef stbIF5
23189+#undef stbIF6
23190+#undef stbIF7
23191+#undef STB_IMAGE_RESIZE_DO_VERTICALS
23192+#undef STBIR__vertical_channels
23193+#undef STB_IMAGE_RESIZE_DO_HORIZONTALS
23194+#undef STBIR_strs_join24
23195+#undef STBIR_strs_join14
23196+#undef STBIR_chans
23197+#ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
23198+#undef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
23199+#endif
23200+
23201+#else // !STB_IMAGE_RESIZE_DO_VERTICALS
23202+
23203+#define STBIR_chans( start, end ) STBIR_strs_join1(start,STBIR__horizontal_channels,end)
23204+
23205+#ifndef stbir__2_coeff_only
23206+#define stbir__2_coeff_only() \
23207+ stbir__1_coeff_only(); \
23208+ stbir__1_coeff_remnant(1);
23209+#endif
23210+
23211+#ifndef stbir__2_coeff_remnant
23212+#define stbir__2_coeff_remnant( ofs ) \
23213+ stbir__1_coeff_remnant(ofs); \
23214+ stbir__1_coeff_remnant((ofs)+1);
23215+#endif
23216+
23217+#ifndef stbir__3_coeff_only
23218+#define stbir__3_coeff_only() \
23219+ stbir__2_coeff_only(); \
23220+ stbir__1_coeff_remnant(2);
23221+#endif
23222+
23223+#ifndef stbir__3_coeff_remnant
23224+#define stbir__3_coeff_remnant( ofs ) \
23225+ stbir__2_coeff_remnant(ofs); \
23226+ stbir__1_coeff_remnant((ofs)+2);
23227+#endif
23228+
23229+#ifndef stbir__3_coeff_setup
23230+#define stbir__3_coeff_setup()
23231+#endif
23232+
23233+#ifndef stbir__4_coeff_start
23234+#define stbir__4_coeff_start() \
23235+ stbir__2_coeff_only(); \
23236+ stbir__2_coeff_remnant(2);
23237+#endif
23238+
23239+#ifndef stbir__4_coeff_continue_from_4
23240+#define stbir__4_coeff_continue_from_4( ofs ) \
23241+ stbir__2_coeff_remnant(ofs); \
23242+ stbir__2_coeff_remnant((ofs)+2);
23243+#endif
23244+
23245+#ifndef stbir__store_output_tiny
23246+#define stbir__store_output_tiny stbir__store_output
23247+#endif
23248+
23249+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_1_coeff)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23250+{
23251+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23252+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23253+ STBIR_SIMD_NO_UNROLL_LOOP_START
23254+ do {
23255+ float const * decode = decode_buffer;
23256+ float const * hc = column->coefficients;
23257+ stbir__1_coeff_only();
23258+ stbir__store_output_tiny();
23259+ } while ( output < output_end );
23260+}
23261+
23262+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_2_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23263+{
23264+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23265+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23266+ #ifdef stbir__x4_start
23267+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23268+ while ( ( output_end - output ) >= 4 )
23269+ {
23270+ stbir__x4_setup();
23271+ stbir__x4_2taps();
23272+ stbir__x4_store();
23273+ }
23274+ if ( output < output_end )
23275+ #endif
23276+ STBIR_SIMD_NO_UNROLL_LOOP_START
23277+ do {
23278+ float const * decode = decode_buffer;
23279+ float const * hc = column->coefficients;
23280+ stbir__2_coeff_only();
23281+ stbir__store_output_tiny();
23282+ } while ( output < output_end );
23283+}
23284+
23285+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_3_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23286+{
23287+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23288+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23289+ #ifdef stbir__x4_start
23290+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23291+ while ( ( output_end - output ) >= 4 )
23292+ {
23293+ stbir__x4_setup();
23294+ stbir__x4_start();
23295+ stbir__x4_finish3();
23296+ stbir__x4_store();
23297+ }
23298+ if ( output < output_end )
23299+ #endif
23300+ STBIR_SIMD_NO_UNROLL_LOOP_START
23301+ do {
23302+ float const * decode = decode_buffer;
23303+ float const * hc = column->coefficients;
23304+ stbir__3_coeff_only();
23305+ stbir__store_output_tiny();
23306+ } while ( output < output_end );
23307+}
23308+
23309+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_4_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23310+{
23311+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23312+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23313+ #ifdef stbir__x4_start
23314+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23315+ while ( ( output_end - output ) >= 4 )
23316+ {
23317+ stbir__x4_setup();
23318+ stbir__x4_start();
23319+ stbir__x4_finish();
23320+ stbir__x4_store();
23321+ }
23322+ if ( output < output_end )
23323+ #endif
23324+ STBIR_SIMD_NO_UNROLL_LOOP_START
23325+ do {
23326+ float const * decode = decode_buffer;
23327+ float const * hc = column->coefficients;
23328+ stbir__4_coeff_start();
23329+ stbir__store_output();
23330+ } while ( output < output_end );
23331+}
23332+
23333+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_5_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23334+{
23335+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23336+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23337+ #ifdef stbir__x4_start
23338+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23339+ while ( ( output_end - output ) >= 4 )
23340+ {
23341+ stbir__x4_setup();
23342+ stbir__x4_start();
23343+ stbir__x4_rem1(4);
23344+ stbir__x4_finish();
23345+ stbir__x4_store();
23346+ }
23347+ if ( output < output_end )
23348+ #endif
23349+ STBIR_SIMD_NO_UNROLL_LOOP_START
23350+ do {
23351+ float const * decode = decode_buffer;
23352+ float const * hc = column->coefficients;
23353+ stbir__4_coeff_start();
23354+ stbir__1_coeff_remnant(4);
23355+ stbir__store_output();
23356+ } while ( output < output_end );
23357+}
23358+
23359+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_6_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23360+{
23361+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23362+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23363+ #ifdef stbir__x4_start
23364+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23365+ while ( ( output_end - output ) >= 4 )
23366+ {
23367+ stbir__x4_setup();
23368+ stbir__x4_start();
23369+ stbir__x4_rem2(4);
23370+ stbir__x4_finish();
23371+ stbir__x4_store();
23372+ }
23373+ if ( output < output_end )
23374+ #endif
23375+ STBIR_SIMD_NO_UNROLL_LOOP_START
23376+ do {
23377+ float const * decode = decode_buffer;
23378+ float const * hc = column->coefficients;
23379+ stbir__4_coeff_start();
23380+ stbir__2_coeff_remnant(4);
23381+ stbir__store_output();
23382+ } while ( output < output_end );
23383+}
23384+
23385+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_7_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23386+{
23387+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23388+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23389+ stbir__3_coeff_setup();
23390+ #ifdef stbir__x4_start
23391+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23392+ while ( ( output_end - output ) >= 4 )
23393+ {
23394+ stbir__x4_setup();
23395+ stbir__x4_start();
23396+ stbir__x4_rem3(4);
23397+ stbir__x4_finish();
23398+ stbir__x4_store();
23399+ }
23400+ if ( output < output_end )
23401+ #endif
23402+ STBIR_SIMD_NO_UNROLL_LOOP_START
23403+ do {
23404+ float const * decode = decode_buffer;
23405+ float const * hc = column->coefficients;
23406+
23407+ stbir__4_coeff_start();
23408+ stbir__3_coeff_remnant(4);
23409+ stbir__store_output();
23410+ } while ( output < output_end );
23411+}
23412+
23413+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_8_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23414+{
23415+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23416+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23417+ #ifdef stbir__x4_start
23418+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23419+ while ( ( output_end - output ) >= 4 )
23420+ {
23421+ stbir__x4_setup();
23422+ stbir__x4_start();
23423+ stbir__x4_cont(4);
23424+ stbir__x4_finish();
23425+ stbir__x4_store();
23426+ }
23427+ if ( output < output_end )
23428+ #endif
23429+ STBIR_SIMD_NO_UNROLL_LOOP_START
23430+ do {
23431+ float const * decode = decode_buffer;
23432+ float const * hc = column->coefficients;
23433+ stbir__4_coeff_start();
23434+ stbir__4_coeff_continue_from_4(4);
23435+ stbir__store_output();
23436+ } while ( output < output_end );
23437+}
23438+
23439+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_9_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23440+{
23441+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23442+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23443+ #ifdef stbir__x4_start
23444+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23445+ while ( ( output_end - output ) >= 4 )
23446+ {
23447+ stbir__x4_setup();
23448+ stbir__x4_start();
23449+ stbir__x4_cont(4);
23450+ stbir__x4_rem1(8);
23451+ stbir__x4_finish();
23452+ stbir__x4_store();
23453+ }
23454+ if ( output < output_end )
23455+ #endif
23456+ STBIR_SIMD_NO_UNROLL_LOOP_START
23457+ do {
23458+ float const * decode = decode_buffer;
23459+ float const * hc = column->coefficients;
23460+ stbir__4_coeff_start();
23461+ stbir__4_coeff_continue_from_4(4);
23462+ stbir__1_coeff_remnant(8);
23463+ stbir__store_output();
23464+ } while ( output < output_end );
23465+}
23466+
23467+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_10_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23468+{
23469+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23470+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23471+ #ifdef stbir__x4_start
23472+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23473+ while ( ( output_end - output ) >= 4 )
23474+ {
23475+ stbir__x4_setup();
23476+ stbir__x4_start();
23477+ stbir__x4_cont(4);
23478+ stbir__x4_rem2(8);
23479+ stbir__x4_finish();
23480+ stbir__x4_store();
23481+ }
23482+ if ( output < output_end )
23483+ #endif
23484+ STBIR_SIMD_NO_UNROLL_LOOP_START
23485+ do {
23486+ float const * decode = decode_buffer;
23487+ float const * hc = column->coefficients;
23488+ stbir__4_coeff_start();
23489+ stbir__4_coeff_continue_from_4(4);
23490+ stbir__2_coeff_remnant(8);
23491+ stbir__store_output();
23492+ } while ( output < output_end );
23493+}
23494+
23495+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_11_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23496+{
23497+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23498+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23499+ stbir__3_coeff_setup();
23500+ #ifdef stbir__x4_start
23501+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23502+ while ( ( output_end - output ) >= 4 )
23503+ {
23504+ stbir__x4_setup();
23505+ stbir__x4_start();
23506+ stbir__x4_cont(4);
23507+ stbir__x4_rem3(8);
23508+ stbir__x4_finish();
23509+ stbir__x4_store();
23510+ }
23511+ if ( output < output_end )
23512+ #endif
23513+ STBIR_SIMD_NO_UNROLL_LOOP_START
23514+ do {
23515+ float const * decode = decode_buffer;
23516+ float const * hc = column->coefficients;
23517+ stbir__4_coeff_start();
23518+ stbir__4_coeff_continue_from_4(4);
23519+ stbir__3_coeff_remnant(8);
23520+ stbir__store_output();
23521+ } while ( output < output_end );
23522+}
23523+
23524+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_12_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23525+{
23526+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23527+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23528+ #ifdef stbir__x4_start
23529+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23530+ while ( ( output_end - output ) >= 4 )
23531+ {
23532+ stbir__x4_setup();
23533+ stbir__x4_start();
23534+ stbir__x4_cont(4);
23535+ stbir__x4_cont(8);
23536+ stbir__x4_finish();
23537+ stbir__x4_store();
23538+ }
23539+ if ( output < output_end )
23540+ #endif
23541+ STBIR_SIMD_NO_UNROLL_LOOP_START
23542+ do {
23543+ float const * decode = decode_buffer;
23544+ float const * hc = column->coefficients;
23545+ stbir__4_coeff_start();
23546+ stbir__4_coeff_continue_from_4(4);
23547+ stbir__4_coeff_continue_from_4(8);
23548+ stbir__store_output();
23549+ } while ( output < output_end );
23550+}
23551+
23552+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_13_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23553+{
23554+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23555+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23556+ #ifdef stbir__x4_start
23557+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23558+ while ( ( output_end - output ) >= 4 )
23559+ {
23560+ stbir__x4_setup();
23561+ stbir__x4_start();
23562+ stbir__x4_cont(4);
23563+ stbir__x4_cont(8);
23564+ stbir__x4_rem1(12);
23565+ stbir__x4_finish();
23566+ stbir__x4_store();
23567+ }
23568+ if ( output < output_end )
23569+ #endif
23570+ STBIR_SIMD_NO_UNROLL_LOOP_START
23571+ do {
23572+ float const * decode = decode_buffer;
23573+ float const * hc = column->coefficients;
23574+ stbir__4_coeff_start();
23575+ stbir__4_coeff_continue_from_4(4);
23576+ stbir__4_coeff_continue_from_4(8);
23577+ stbir__1_coeff_remnant(12);
23578+ stbir__store_output();
23579+ } while ( output < output_end );
23580+}
23581+
23582+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_14_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23583+{
23584+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23585+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23586+ #ifdef stbir__x4_start
23587+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23588+ while ( ( output_end - output ) >= 4 )
23589+ {
23590+ stbir__x4_setup();
23591+ stbir__x4_start();
23592+ stbir__x4_cont(4);
23593+ stbir__x4_cont(8);
23594+ stbir__x4_rem2(12);
23595+ stbir__x4_finish();
23596+ stbir__x4_store();
23597+ }
23598+ if ( output < output_end )
23599+ #endif
23600+ STBIR_SIMD_NO_UNROLL_LOOP_START
23601+ do {
23602+ float const * decode = decode_buffer;
23603+ float const * hc = column->coefficients;
23604+ stbir__4_coeff_start();
23605+ stbir__4_coeff_continue_from_4(4);
23606+ stbir__4_coeff_continue_from_4(8);
23607+ stbir__2_coeff_remnant(12);
23608+ stbir__store_output();
23609+ } while ( output < output_end );
23610+}
23611+
23612+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_15_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23613+{
23614+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23615+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23616+ stbir__3_coeff_setup();
23617+ #ifdef stbir__x4_start
23618+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23619+ while ( ( output_end - output ) >= 4 )
23620+ {
23621+ stbir__x4_setup();
23622+ stbir__x4_start();
23623+ stbir__x4_cont(4);
23624+ stbir__x4_cont(8);
23625+ stbir__x4_rem3(12);
23626+ stbir__x4_finish();
23627+ stbir__x4_store();
23628+ }
23629+ if ( output < output_end )
23630+ #endif
23631+ STBIR_SIMD_NO_UNROLL_LOOP_START
23632+ do {
23633+ float const * decode = decode_buffer;
23634+ float const * hc = column->coefficients;
23635+ stbir__4_coeff_start();
23636+ stbir__4_coeff_continue_from_4(4);
23637+ stbir__4_coeff_continue_from_4(8);
23638+ stbir__3_coeff_remnant(12);
23639+ stbir__store_output();
23640+ } while ( output < output_end );
23641+}
23642+
23643+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_16_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23644+{
23645+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23646+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23647+ #ifdef stbir__x4_start
23648+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23649+ while ( ( output_end - output ) >= 4 )
23650+ {
23651+ stbir__x4_setup();
23652+ stbir__x4_start();
23653+ stbir__x4_cont(4);
23654+ stbir__x4_cont(8);
23655+ stbir__x4_cont(12);
23656+ stbir__x4_finish();
23657+ stbir__x4_store();
23658+ }
23659+ if ( output < output_end )
23660+ #endif
23661+ STBIR_SIMD_NO_UNROLL_LOOP_START
23662+ do {
23663+ float const * decode = decode_buffer;
23664+ float const * hc = column->coefficients;
23665+ stbir__4_coeff_start();
23666+ stbir__4_coeff_continue_from_4(4);
23667+ stbir__4_coeff_continue_from_4(8);
23668+ stbir__4_coeff_continue_from_4(12);
23669+ stbir__store_output();
23670+ } while ( output < output_end );
23671+}
23672+
23673+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_17_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23674+{
23675+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23676+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23677+ #ifdef stbir__x4_start
23678+ // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
23679+ while ( ( output_end - output ) >= 4 )
23680+ {
23681+ stbir__x4_setup();
23682+ stbir__x4_start();
23683+ stbir__x4_cont(4);
23684+ stbir__x4_cont(8);
23685+ stbir__x4_cont(12);
23686+ stbir__x4_rem1(16);
23687+ stbir__x4_finish();
23688+ stbir__x4_store();
23689+ }
23690+ if ( output < output_end )
23691+ #endif
23692+ STBIR_SIMD_NO_UNROLL_LOOP_START
23693+ do {
23694+ float const * decode = decode_buffer;
23695+ float const * hc = column->coefficients;
23696+ stbir__4_coeff_start();
23697+ stbir__4_coeff_continue_from_4(4);
23698+ stbir__4_coeff_continue_from_4(8);
23699+ stbir__4_coeff_continue_from_4(12);
23700+ stbir__1_coeff_remnant(16);
23701+ stbir__store_output();
23702+ } while ( output < output_end );
23703+}
23704+
23705+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod0 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23706+{
23707+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23708+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23709+ STBIR_SIMD_NO_UNROLL_LOOP_START
23710+ do {
23711+ float const * decode = decode_buffer;
23712+ int n = ( column->width - 4 + 3 ) >> 2;
23713+ float const * hc = column->coefficients;
23714+
23715+ stbir__4_coeff_start();
23716+ STBIR_SIMD_NO_UNROLL_LOOP_START
23717+ do {
23718+ hc += 4;
23719+ decode += STBIR__horizontal_channels * 4;
23720+ stbir__4_coeff_continue_from_4( 0 );
23721+ --n;
23722+ } while ( n > 0 );
23723+ stbir__store_output();
23724+ } while ( output < output_end );
23725+}
23726+
23727+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod1 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23728+{
23729+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23730+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23731+ STBIR_SIMD_NO_UNROLL_LOOP_START
23732+ do {
23733+ float const * decode = decode_buffer;
23734+ int n = ( column->width - 5 + 3 ) >> 2;
23735+ float const * hc = column->coefficients;
23736+
23737+ stbir__4_coeff_start();
23738+ STBIR_SIMD_NO_UNROLL_LOOP_START
23739+ do {
23740+ hc += 4;
23741+ decode += STBIR__horizontal_channels * 4;
23742+ stbir__4_coeff_continue_from_4( 0 );
23743+ --n;
23744+ } while ( n > 0 );
23745+ stbir__1_coeff_remnant( 4 );
23746+ stbir__store_output();
23747+ } while ( output < output_end );
23748+}
23749+
23750+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod2 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23751+{
23752+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23753+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23754+ STBIR_SIMD_NO_UNROLL_LOOP_START
23755+ do {
23756+ float const * decode = decode_buffer;
23757+ int n = ( column->width - 6 + 3 ) >> 2;
23758+ float const * hc = column->coefficients;
23759+
23760+ stbir__4_coeff_start();
23761+ STBIR_SIMD_NO_UNROLL_LOOP_START
23762+ do {
23763+ hc += 4;
23764+ decode += STBIR__horizontal_channels * 4;
23765+ stbir__4_coeff_continue_from_4( 0 );
23766+ --n;
23767+ } while ( n > 0 );
23768+ stbir__2_coeff_remnant( 4 );
23769+
23770+ stbir__store_output();
23771+ } while ( output < output_end );
23772+}
23773+
23774+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod3 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23775+{
23776+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23777+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23778+ stbir__3_coeff_setup();
23779+ STBIR_SIMD_NO_UNROLL_LOOP_START
23780+ do {
23781+ float const * decode = decode_buffer;
23782+ int n = ( column->width - 7 + 3 ) >> 2;
23783+ float const * hc = column->coefficients;
23784+
23785+ stbir__4_coeff_start();
23786+ STBIR_SIMD_NO_UNROLL_LOOP_START
23787+ do {
23788+ hc += 4;
23789+ decode += STBIR__horizontal_channels * 4;
23790+ stbir__4_coeff_continue_from_4( 0 );
23791+ --n;
23792+ } while ( n > 0 );
23793+ stbir__3_coeff_remnant( 4 );
23794+
23795+ stbir__store_output();
23796+ } while ( output < output_end );
23797+}
23798+
23799+// polyphase period 1 kernels (integer downsamples, see stbir__resample_horizontal_gather): every output uses the same
23800+// coefficient set, so it is loaded into registers once per run and the pixel loop is just the decode loads and
23801+// multiply-adds. the same steps as the kernels above, for the widths integer downsamples produce (4, 8: 2:1 with the
23802+// support 1 and 2 filters; 5, 9: the same one tap wider; 11/12/13 and 16/17: 3:1 and 4:1 with the support 2 filters -
23803+// 3:1 trims to 11 because its outermost taps land exactly on the filter's zero at +-2).
23804+// only for the channel counts that define the step macros.
23805+#ifdef stbir__period1_setup_4
23806+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_4_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23807+{
23808+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23809+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23810+ float const * hc = column->coefficients;
23811+ int advance = column->advance;
23812+ stbir__period1_setup_4()
23813+ STBIR__UNUSED( entry_stride );
23814+ #ifdef stbir__period1_x4_start
23815+ // four outputs at a time while they last (one channel only)
23816+ while ( ( output_end - output ) >= 4 )
23817+ {
23818+ stbir__period1_x4_setup();
23819+ stbir__period1_x4_start();
23820+ stbir__period1_x4_store();
23821+ }
23822+ if ( output < output_end )
23823+ #endif
23824+ STBIR_SIMD_NO_UNROLL_LOOP_START
23825+ do {
23826+ float const * decode = decode_buffer;
23827+ stbir__period1_4_coeff_start();
23828+ stbir__period1_store_output();
23829+ } while ( output < output_end );
23830+}
23831+
23832+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_5_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23833+{
23834+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23835+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23836+ float const * hc = column->coefficients;
23837+ int advance = column->advance;
23838+ stbir__period1_setup_5()
23839+ STBIR__UNUSED( entry_stride );
23840+ #ifdef stbir__period1_x4_start
23841+ // four outputs at a time while they last (one channel only)
23842+ while ( ( output_end - output ) >= 4 )
23843+ {
23844+ stbir__period1_x4_setup();
23845+ stbir__period1_x4_start();
23846+ stbir__period1_x4_remnant(4);
23847+ stbir__period1_x4_store();
23848+ }
23849+ if ( output < output_end )
23850+ #endif
23851+ STBIR_SIMD_NO_UNROLL_LOOP_START
23852+ do {
23853+ float const * decode = decode_buffer;
23854+ stbir__period1_4_coeff_start();
23855+ stbir__period1_1_coeff_remnant(4);
23856+ stbir__period1_store_output();
23857+ } while ( output < output_end );
23858+}
23859+
23860+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_8_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23861+{
23862+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23863+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23864+ float const * hc = column->coefficients;
23865+ int advance = column->advance;
23866+ stbir__period1_setup_8()
23867+ STBIR__UNUSED( entry_stride );
23868+ #ifdef stbir__period1_x4_start
23869+ // four outputs at a time while they last (one channel only)
23870+ while ( ( output_end - output ) >= 4 )
23871+ {
23872+ stbir__period1_x4_setup();
23873+ stbir__period1_x4_start();
23874+ stbir__period1_x4_continue();
23875+ stbir__period1_x4_store();
23876+ }
23877+ if ( output < output_end )
23878+ #endif
23879+ STBIR_SIMD_NO_UNROLL_LOOP_START
23880+ do {
23881+ float const * decode = decode_buffer;
23882+ stbir__period1_4_coeff_start();
23883+ stbir__period1_4_coeff_continue();
23884+ stbir__period1_store_output();
23885+ } while ( output < output_end );
23886+}
23887+
23888+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_9_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23889+{
23890+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23891+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23892+ float const * hc = column->coefficients;
23893+ int advance = column->advance;
23894+ stbir__period1_setup_9()
23895+ STBIR__UNUSED( entry_stride );
23896+ #ifdef stbir__period1_x4_start
23897+ // four outputs at a time while they last (one channel only)
23898+ while ( ( output_end - output ) >= 4 )
23899+ {
23900+ stbir__period1_x4_setup();
23901+ stbir__period1_x4_start();
23902+ stbir__period1_x4_continue();
23903+ stbir__period1_x4_remnant(8);
23904+ stbir__period1_x4_store();
23905+ }
23906+ if ( output < output_end )
23907+ #endif
23908+ STBIR_SIMD_NO_UNROLL_LOOP_START
23909+ do {
23910+ float const * decode = decode_buffer;
23911+ stbir__period1_4_coeff_start();
23912+ stbir__period1_4_coeff_continue();
23913+ stbir__period1_1_coeff_remnant(8);
23914+ stbir__period1_store_output();
23915+ } while ( output < output_end );
23916+}
23917+
23918+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_11_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23919+{
23920+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23921+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23922+ float const * hc = column->coefficients;
23923+ int advance = column->advance;
23924+ stbir__period1_setup_11()
23925+ STBIR__UNUSED( entry_stride );
23926+ #ifdef stbir__period1_x4_start
23927+ // four outputs at a time while they last (one channel only)
23928+ while ( ( output_end - output ) >= 4 )
23929+ {
23930+ stbir__period1_x4_setup();
23931+ stbir__period1_x4_start();
23932+ stbir__period1_x4_continue();
23933+ stbir__period1_x4_remnant3(8);
23934+ stbir__period1_x4_store();
23935+ }
23936+ if ( output < output_end )
23937+ #endif
23938+ STBIR_SIMD_NO_UNROLL_LOOP_START
23939+ do {
23940+ float const * decode = decode_buffer;
23941+ stbir__period1_4_coeff_start();
23942+ stbir__period1_4_coeff_continue();
23943+ stbir__period1_3_coeff_remnant(8);
23944+ stbir__period1_store_output();
23945+ } while ( output < output_end );
23946+}
23947+
23948+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_12_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23949+{
23950+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23951+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23952+ float const * hc = column->coefficients;
23953+ int advance = column->advance;
23954+ stbir__period1_setup_12()
23955+ STBIR__UNUSED( entry_stride );
23956+ #ifdef stbir__period1_x4_start
23957+ // four outputs at a time while they last (one channel only)
23958+ while ( ( output_end - output ) >= 4 )
23959+ {
23960+ stbir__period1_x4_setup();
23961+ stbir__period1_x4_start();
23962+ stbir__period1_x4_continue();
23963+ stbir__period1_x4_continue2();
23964+ stbir__period1_x4_store();
23965+ }
23966+ if ( output < output_end )
23967+ #endif
23968+ STBIR_SIMD_NO_UNROLL_LOOP_START
23969+ do {
23970+ float const * decode = decode_buffer;
23971+ stbir__period1_4_coeff_start();
23972+ stbir__period1_4_coeff_continue();
23973+ stbir__period1_4_coeff_continue2();
23974+ stbir__period1_store_output();
23975+ } while ( output < output_end );
23976+}
23977+
23978+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_13_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
23979+{
23980+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
23981+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
23982+ float const * hc = column->coefficients;
23983+ int advance = column->advance;
23984+ stbir__period1_setup_13()
23985+ STBIR__UNUSED( entry_stride );
23986+ #ifdef stbir__period1_x4_start
23987+ // four outputs at a time while they last (one channel only)
23988+ while ( ( output_end - output ) >= 4 )
23989+ {
23990+ stbir__period1_x4_setup();
23991+ stbir__period1_x4_start();
23992+ stbir__period1_x4_continue();
23993+ stbir__period1_x4_continue2();
23994+ stbir__period1_x4_remnant(12);
23995+ stbir__period1_x4_store();
23996+ }
23997+ if ( output < output_end )
23998+ #endif
23999+ STBIR_SIMD_NO_UNROLL_LOOP_START
24000+ do {
24001+ float const * decode = decode_buffer;
24002+ stbir__period1_4_coeff_start();
24003+ stbir__period1_4_coeff_continue();
24004+ stbir__period1_4_coeff_continue2();
24005+ stbir__period1_1_coeff_remnant(12);
24006+ stbir__period1_store_output();
24007+ } while ( output < output_end );
24008+}
24009+
24010+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_16_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
24011+{
24012+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
24013+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
24014+ float const * hc = column->coefficients;
24015+ int advance = column->advance;
24016+ stbir__period1_setup_16()
24017+ STBIR__UNUSED( entry_stride );
24018+ #ifdef stbir__period1_x4_start
24019+ // four outputs at a time while they last (one channel only)
24020+ while ( ( output_end - output ) >= 4 )
24021+ {
24022+ stbir__period1_x4_setup();
24023+ stbir__period1_x4_start();
24024+ stbir__period1_x4_continue();
24025+ stbir__period1_x4_continue2();
24026+ stbir__period1_x4_continue3();
24027+ stbir__period1_x4_store();
24028+ }
24029+ if ( output < output_end )
24030+ #endif
24031+ STBIR_SIMD_NO_UNROLL_LOOP_START
24032+ do {
24033+ float const * decode = decode_buffer;
24034+ stbir__period1_4_coeff_start();
24035+ stbir__period1_4_coeff_continue();
24036+ stbir__period1_4_coeff_continue2();
24037+ stbir__period1_4_coeff_continue3();
24038+ stbir__period1_store_output();
24039+ } while ( output < output_end );
24040+}
24041+
24042+static void STBIR_chans( stbir__horizontal_gather_,_channels_with_17_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
24043+{
24044+ float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
24045+ float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
24046+ float const * hc = column->coefficients;
24047+ int advance = column->advance;
24048+ stbir__period1_setup_17()
24049+ STBIR__UNUSED( entry_stride );
24050+ #ifdef stbir__period1_x4_start
24051+ // four outputs at a time while they last (one channel only)
24052+ while ( ( output_end - output ) >= 4 )
24053+ {
24054+ stbir__period1_x4_setup();
24055+ stbir__period1_x4_start();
24056+ stbir__period1_x4_continue();
24057+ stbir__period1_x4_continue2();
24058+ stbir__period1_x4_continue3();
24059+ stbir__period1_x4_remnant(16);
24060+ stbir__period1_x4_store();
24061+ }
24062+ if ( output < output_end )
24063+ #endif
24064+ STBIR_SIMD_NO_UNROLL_LOOP_START
24065+ do {
24066+ float const * decode = decode_buffer;
24067+ stbir__period1_4_coeff_start();
24068+ stbir__period1_4_coeff_continue();
24069+ stbir__period1_4_coeff_continue2();
24070+ stbir__period1_4_coeff_continue3();
24071+ stbir__period1_1_coeff_remnant(16);
24072+ stbir__period1_store_output();
24073+ } while ( output < output_end );
24074+}
24075+
24076+// by coefficient count (null = use the normal kernel over the one entry)
24077+static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_period1_funcs)[18]=
24078+{
24079+ 0, 0, 0, 0,
24080+ STBIR_chans(stbir__horizontal_gather_,_channels_with_4_coeffs_period1),
24081+ STBIR_chans(stbir__horizontal_gather_,_channels_with_5_coeffs_period1),
24082+ 0, 0,
24083+ STBIR_chans(stbir__horizontal_gather_,_channels_with_8_coeffs_period1),
24084+ STBIR_chans(stbir__horizontal_gather_,_channels_with_9_coeffs_period1),
24085+ 0,
24086+ STBIR_chans(stbir__horizontal_gather_,_channels_with_11_coeffs_period1),
24087+ STBIR_chans(stbir__horizontal_gather_,_channels_with_12_coeffs_period1),
24088+ STBIR_chans(stbir__horizontal_gather_,_channels_with_13_coeffs_period1),
24089+ 0, 0,
24090+ STBIR_chans(stbir__horizontal_gather_,_channels_with_16_coeffs_period1),
24091+ STBIR_chans(stbir__horizontal_gather_,_channels_with_17_coeffs_period1)
24092+};
24093+#else
24094+static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_period1_funcs)[18]= { 0 };
24095+#endif
24096+
24097+static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_funcs)[4]=
24098+{
24099+ STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod0),
24100+ STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod1),
24101+ STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod2),
24102+ STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod3),
24103+};
24104+
24105+static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_funcs)[17]=
24106+{
24107+ STBIR_chans(stbir__horizontal_gather_,_channels_with_1_coeff),
24108+ STBIR_chans(stbir__horizontal_gather_,_channels_with_2_coeffs),
24109+ STBIR_chans(stbir__horizontal_gather_,_channels_with_3_coeffs),
24110+ STBIR_chans(stbir__horizontal_gather_,_channels_with_4_coeffs),
24111+ STBIR_chans(stbir__horizontal_gather_,_channels_with_5_coeffs),
24112+ STBIR_chans(stbir__horizontal_gather_,_channels_with_6_coeffs),
24113+ STBIR_chans(stbir__horizontal_gather_,_channels_with_7_coeffs),
24114+ STBIR_chans(stbir__horizontal_gather_,_channels_with_8_coeffs),
24115+ STBIR_chans(stbir__horizontal_gather_,_channels_with_9_coeffs),
24116+ STBIR_chans(stbir__horizontal_gather_,_channels_with_10_coeffs),
24117+ STBIR_chans(stbir__horizontal_gather_,_channels_with_11_coeffs),
24118+ STBIR_chans(stbir__horizontal_gather_,_channels_with_12_coeffs),
24119+ STBIR_chans(stbir__horizontal_gather_,_channels_with_13_coeffs),
24120+ STBIR_chans(stbir__horizontal_gather_,_channels_with_14_coeffs),
24121+ STBIR_chans(stbir__horizontal_gather_,_channels_with_15_coeffs),
24122+ STBIR_chans(stbir__horizontal_gather_,_channels_with_16_coeffs),
24123+ STBIR_chans(stbir__horizontal_gather_,_channels_with_17_coeffs),
24124+};
24125+
24126+#ifdef stbir__period1_setup_4
24127+#undef stbir__period1_setup_4
24128+#undef stbir__period1_setup_5
24129+#undef stbir__period1_setup_8
24130+#undef stbir__period1_setup_9
24131+#undef stbir__period1_setup_11
24132+#undef stbir__period1_setup_12
24133+#undef stbir__period1_setup_13
24134+#undef stbir__period1_setup_16
24135+#undef stbir__period1_setup_17
24136+#undef stbir__period1_4_coeff_start
24137+#undef stbir__period1_4_coeff_continue
24138+#undef stbir__period1_4_coeff_continue2
24139+#undef stbir__period1_4_coeff_continue3
24140+#undef stbir__period1_1_coeff_remnant
24141+#undef stbir__period1_3_coeff_remnant
24142+#undef stbir__period1_store_output
24143+#endif
24144+#ifdef stbir__period1_x4_start
24145+#undef stbir__period1_x4_setup
24146+#undef stbir__period1_x4_start
24147+#undef stbir__period1_x4_next
24148+#undef stbir__period1_x4_continue
24149+#undef stbir__period1_x4_continue2
24150+#undef stbir__period1_x4_continue3
24151+#undef stbir__period1_x4_remnant
24152+#undef stbir__period1_x4_remnant3
24153+#undef stbir__period1_x4_store
24154+#endif
24155+#undef STBIR__horizontal_channels
24156+#undef STB_IMAGE_RESIZE_DO_HORIZONTALS
24157+#ifdef stbir__x4_start
24158+#undef stbir__x4_setup
24159+#undef stbir__x4_start
24160+#undef stbir__x4_cont
24161+#undef stbir__x4_rem1
24162+#undef stbir__x4_rem2
24163+#undef stbir__x4_rem3
24164+#undef stbir__x4_2taps
24165+#undef stbir__x4_finish
24166+#undef stbir__x4_finish3
24167+#undef stbir__x4_store
24168+#endif
24169+#undef stbir__1_coeff_only
24170+#undef stbir__1_coeff_remnant
24171+#undef stbir__2_coeff_only
24172+#undef stbir__2_coeff_remnant
24173+#undef stbir__3_coeff_only
24174+#undef stbir__3_coeff_remnant
24175+#undef stbir__3_coeff_setup
24176+#undef stbir__4_coeff_start
24177+#undef stbir__4_coeff_continue_from_4
24178+#undef stbir__store_output
24179+#undef stbir__store_output_tiny
24180+#undef STBIR_chans
24181+
24182+#endif // HORIZONALS
24183+
24184+#undef STBIR_strs_join2
24185+#undef STBIR_strs_join1
24186+
24187+#endif // STB_IMAGE_RESIZE_DO_HORIZONTALS/VERTICALS/CODERS
24188+
24189+/*
24190+------------------------------------------------------------------------------
24191+This software is available under 2 licenses -- choose whichever you prefer.
24192+------------------------------------------------------------------------------
24193+ALTERNATIVE A - MIT License
24194+Copyright (c) 2017 Sean Barrett
24195+Permission is hereby granted, free of charge, to any person obtaining a copy of
24196+this software and associated documentation files (the "Software"), to deal in
24197+the Software without restriction, including without limitation the rights to
24198+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
24199+of the Software, and to permit persons to whom the Software is furnished to do
24200+so, subject to the following conditions:
24201+The above copyright notice and this permission notice shall be included in all
24202+copies or substantial portions of the Software.
24203+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
24204+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
24205+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
24206+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
24207+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
24208+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
24209+SOFTWARE.
24210+------------------------------------------------------------------------------
24211+ALTERNATIVE B - Public Domain (www.unlicense.org)
24212+This is free and unencumbered software released into the public domain.
24213+Anyone is free to copy, modify, publish, use, compile, sell, or distribute this
24214+software, either in source code form or as a compiled binary, for any purpose,
24215+commercial or non-commercial, and by any means.
24216+In jurisdictions that recognize copyright laws, the author or authors of this
24217+software dedicate any and all copyright interest in the software to the public
24218+domain. We make this dedication for the benefit of the public at large and to
24219+the detriment of our heirs and successors. We intend this dedication to be an
24220+overt act of relinquishment in perpetuity of all present and future rights to
24221+this software under copyright law.
24222+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
24223+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
24224+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
24225+AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
24226+ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
24227+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
24228+------------------------------------------------------------------------------
24229+*/