f35caleb/chesco
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Complete residential site build with Docker Compose & Gitea Actions deployment

Browse Source
This commit is contained in:
CalebLewallen
2026-05-13 17:31:20 -04:00
parent f9c4e13a50
commit 3c22823f72
19354 changed files with 2097331 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
node_modules/sitemap/CHANGELOG.md
Generated Vendored
+586
View File
@@ -0,0 +1,586 @@
# Changelog
## 9.0.1 — Security Patch
- **BB-01**: Fix XML injection via unescaped `xslUrl` in stylesheet processing instruction — special characters (`&`, `"`, `<`, `>`) in the XSL URL are now escaped before being interpolated into the `<?xml-stylesheet?>` processing instruction
- **BB-02**: Enforce 50,000 URL hard limit in `XMLToSitemapItemStream` — the parser now stops emitting items and emits an error when the limit is exceeded, rather than merely logging a warning
- **BB-03**: Cap parser error array at 100 entries to prevent memory DoS — `XMLToSitemapItemStream` now tracks a separate `errorCount` and stops appending to the `errors` array beyond `LIMITS.MAX_PARSER_ERRORS`
- **BB-04**: Reject absolute `destinationDir` paths in `simpleSitemapAndIndex` to prevent arbitrary file writes — passing an absolute path (e.g. `/tmp/sitemaps`) now throws immediately with a descriptive error
- **BB-05**: `parseSitemapIndex` now destroys source and parser streams immediately when the `maxEntries` limit is exceeded, preventing unbounded memory consumption from large sitemap index files
## 9.0.0 - 2025-11-01
This major release modernizes the package with ESM-first architecture, drops support for Node.js < 20, and includes comprehensive security and robustness improvements.
### [BREAKING CHANGES]
#### Dropped Node.js < 20 Support
- **Node.js >=20.19.5 now required** (previously >=14.0.0)
- **npm >=10.8.2 now required** (previously >=6.0.0)
- Dropped support for Node.js 14, 16, and 18
#### ESM Conversion with Dual Package Support
- Package now uses `"type": "module"` in package.json
- Built as dual ESM/CJS package with conditional exports
- **Import paths in ESM require `.js` extensions** (TypeScript will add these automatically)
- Both ESM and CommonJS imports continue to work:
```js
// ESM (new default)
import { SitemapStream } from 'sitemap'
// CommonJS (still supported)
const { SitemapStream } = require('sitemap')
```
- CLI remains ESM-only at `dist/esm/cli.js`
#### Build Output Changes
- ESM output: `dist/esm/` (was `dist/`)
- CJS output: `dist/cjs/` (new)
- TypeScript definitions: `dist/esm/index.d.ts` (was `dist/index.d.ts`)
#### Node.js Modernization
- All built-in Node.js modules now use `node:` protocol imports (`node:stream`, `node:fs`, etc.)
- Uses native promise-based `pipeline` from `node:stream/promises` (instead of `promisify(pipeline)`)
- TypeScript target updated to ES2023 (from ES2022)
### New Exports
The following validation functions and constants are now part of the public API:
**Validation Functions** (from `lib/validation.js`):
- `validateURL()`, `validatePath()`, `validateLimit()`, `validatePublicBasePath()`, `validateXSLUrl()`
- Type guards: `isPriceType()`, `isResolution()`, `isValidChangeFreq()`, `isValidYesNo()`, `isAllowDeny()`
- `validators` - object containing regex validators for all sitemap fields
**Constants** (from `lib/constants.js`):
- `LIMITS` - security limits object (max URL length, max items per sitemap, video/news/image constraints, etc.)
- `DEFAULT_SITEMAP_ITEM_LIMIT` - default items per sitemap file (45,000)
**New Type Export**:
- `SimpleSitemapAndIndexOptions` interface now exported
### Features
#### Comprehensive Security Validation
- **Parser Security** (#461): Added resource limits and comprehensive validation to sitemap index parser and stream
- Max 50K URLs per sitemap, 1K images, 100 videos per entry
- String length limits on all fields
- URL validation (http/https only, max 2048 chars)
- Protocol injection prevention (blocks javascript:, data:, file:, ftp:)
- Path traversal prevention (blocks `..` sequences)
- **Stream Validation** (#456, #455, #454): Added comprehensive validation to all stream classes
- Enhanced XML entity escaping (including `>` character)
- Attribute name validation
- Date format validation (ISO 8601)
- Input validation for numbers (reject NaN/Infinity), dates (check Invalid Date)
- XSL URL validation to prevent script injection
- Custom namespace validation (max 20 namespaces, max 512 chars each)
- **XML Generation Security** (#457): Comprehensive validation and documentation in sitemap-xml
- Safe XML attribute and element generation
- Protection against XML injection attacks
#### Robustness Improvements
- **Sitemap Item Stream** (#453): Improved robustness and type safety
- **Sitemap Index Stream** (#449): Enhanced robustness and test coverage
- **Sitemap Index Parser** (#448): Improved error handling and robustness
- **Code Quality** (#458): Comprehensive security and code quality improvements across codebase
### Fixes
- Fixed TS151002 warning and test race condition (#455)
- Improved sitemap-item-stream robustness and type safety (#453)
- Enhanced sitemap-index-stream error handling (#449)
- Improved sitemap-index-parser error handling (#448)
- Fixed coverage reporting (#399, #434)
- Fixed invalid XML regex for better performance (#437, #417)
- Improved normalizeURL performance (#416)
### Refactoring
- **Architecture Reorganization** (#460): Consolidated constants and validation
- Created `lib/constants.ts` - single source of truth for all shared constants
- Created `lib/validation.ts` - centralized all validation logic and type guards
- Eliminated duplicate constants and validation code across files
- Prevents inconsistencies where different files used different values
### Infrastructure
#### Build System
- Dual ESM/CJS build with separate TypeScript configurations
- `tsconfig.json` - ESM build (NodeNext module resolution)
- `tsconfig.cjs.json` - CJS build (CommonJS module)
- Build outputs `package.json` with `"type": "commonjs"` to `dist/cjs/`
- Test infrastructure converted to ESM
- Updated Jest configuration for ESM support
#### Testing
- Converted to ts-jest for better TypeScript support (#434)
- All 172+ tests passing with 91%+ code coverage
- Enhanced security-focused test coverage
- Performance tests converted to `.mjs` format
#### Dependencies
- Updated `sax` from ^1.2.4 to ^1.4.1
- Updated `@types/node` from ^17.0.5 to ^24.7.2
- Removed unused dependencies (#459)
- Updated all dev dependencies to latest versions
- Replaced babel-based test setup with ts-jest
#### Developer Experience
- Updated examples to ESM syntax in README (#452)
- Updated API documentation for accuracy and ESM syntax (#452)
- Added comprehensive CLAUDE.md with architecture documentation
- Improved ESLint and Prettier integration
- Updated git hooks with Husky 9.x
### Upgrade Guide for 9.0.0
#### 1. Update Node.js Version
Ensure you are running Node.js >=20.19.5 and npm >=10.8.2:
```bash
node --version # Should be 20.19.5 or higher
npm --version # Should be 10.8.2 or higher
```
#### 2. Update Package
```bash
npm install sitemap@9.0.0
```
#### 3. Import Syntax (No Changes Required for Most Users)
Both ESM and CommonJS imports continue to work:
```js
// ESM - works the same as before
import { SitemapStream, streamToPromise } from 'sitemap'
// CommonJS - works the same as before
const { SitemapStream, streamToPromise } = require('sitemap')
```
**Note**: If you're importing from the package in an ESM context, the module resolution happens automatically. If you're directly importing library files (not recommended), you'll need `.js` extensions.
#### 4. Existing Code Compatibility
- ✅ **All existing valid data continues to work unchanged**
- ✅ **Public API is fully compatible** - same classes, methods, and options
- ✅ **Stream behavior unchanged** - all streaming patterns continue to work
- ✅ **Error handling unchanged** - `ErrorLevel.WARN` default behavior maintained
- ⚠️ **Invalid data may now be rejected** due to enhanced security validation
- URLs must be http/https protocol (no javascript:, data:, etc.)
- String lengths enforced per sitemaps.org spec
- Resource limits enforced (50K URLs, 1K images, 100 videos per entry)
#### 5. TypeScript Users
- Update `tsconfig.json` if needed to support ES2023
- Type definitions are now at `dist/esm/index.d.ts` (automatically resolved by package.json exports)
- No changes needed to your TypeScript code
#### 6. New Optional Features
You can now import validation utilities and constants if needed:
```js
import { LIMITS, validateURL, validators } from 'sitemap'
// Check limits
console.log(LIMITS.MAX_URL_LENGTH) // 2048
// Validate URLs
const url = validateURL('https://example.com/page')
// Use validators
if (validators['video:rating'].test('4.5')) {
// valid rating
}
```
## 8.0.2 - Bug Fix Release
### Bug Fixes
- **fix #464**: Support `xsi:schemaLocation` in custom namespaces - thanks @dzakki
- Extended custom namespace validation to accept namespace-qualified attributes (like `xsi:schemaLocation`) in addition to `xmlns` declarations
- The validation regex now matches both `xmlns:prefix="uri"` and `prefix:attribute="value"` patterns
- Enables proper W3C schema validation while maintaining security validation for malicious content
- Added comprehensive tests including security regression tests
### Example Usage
The following now works correctly (as documented in README):
```javascript
const sms = new SitemapStream({
xmlns: {
custom: [
'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"',
'xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"'
]
}
});
```
### Testing
- ✅ All existing tests passing
- ✅ 8 new tests added covering positive and security scenarios
- ✅ 100% backward compatible with 8.0.1
### Files Changed
2 files changed: 144 insertions, 5 deletions
## 8.0.1 - Security Patch Release
**SECURITY FIXES** - This release backports comprehensive security patches from 9.0.0 to 8.0.x
### Security Improvements
- **XML Injection Prevention**: Enhanced XML entity escaping, added `>` character escaping, attribute name validation
- **Parser Security**: Added resource limits (max 50K URLs, 1K images, 100 videos per sitemap), string length limits, URL validation (http/https only, max 2048 chars)
- **Protocol Injection Prevention**: Block dangerous protocols (javascript:, data:, file:, ftp:) in sitemap index parser
- **DoS Protection**: Memory exhaustion protection, URL length validation, date format validation (ISO 8601)
- **Path Traversal Prevention**: Block `..` sequences in file paths
- **Command Injection Fix**: xmllint now uses stdin exclusively instead of file paths
- **Input Validation**: Comprehensive validation for all user inputs - numbers (reject NaN/Infinity), dates (check Invalid Date), URLs, paths
- **XSS Prevention**: XSL URL validation to prevent script injection
- **Namespace Security**: Custom namespace validation (max 20, max 512 chars each)
### Infrastructure
- Added `lib/constants.ts` - Centralized security limits and constants
- Added `lib/validation.ts` - Comprehensive validation functions
- Added new security-related error classes
### Backward Compatibility
- ✅ **100% API compatible** with 8.0.0
- Added `XMLToSitemapItemStream.error` getter for backward compatibility (returns `errors[0]`)
- All existing valid inputs continue to work
- Only rejects invalid/malicious inputs
- Default `ErrorLevel.WARN` behavior unchanged
### Dependencies Updated
- `sax`: ^1.2.4 → ^1.4.1 (security updates)
### Files Changed
17 files changed: 2,122 additions, 245 deletions
### Testing
- All 94 existing tests passing
- No breaking changes to public API
## 8.0.0
- fix #423 via #424 thanks @huntharo - Propagate errors in SitemapAndIndexStream
- drop node 12 support
## 7.1.2
- fix #425 via #426 thanks to @huntharo update streamToPromise to bubble up errors + jsDoc
- fix #415 thanks to @mohd-akram Fix circular dependency breaking Node.js 20.6
- non-breaking updates of dependent packages
## 7.1.1
- fix #378 exit code not set on parse failure. A proper error will be set on the stream now.
- fix #384 thanks @tomcek112 parseSitemapIndex not included in 7.1.0 release
- fix #356 thanks @vandres - SitemapIndexStream now has lastmodDateOnly
- Fix #375 thanks @huntharo parseSitemap and parseSitemapIndex uncatchable errors
- Filter out null as well when writing XML thanks @huntharo #376
## 7.1.0
- bumped types dependency for node
- bumped all dev dependencies - includes some prettier changes
- package-lock updated to version 2
## 7.0.0
### [BREAKING]
- dropped support for Node 10, added support for Node 16
- removed deprecated createSitemapsAndIndex. use SitemapAndIndexStream or simpleSitemapAndIndex
- dropped deprecated `getSitemapStream` option for SitemapAndIndexStream that does not return a write stream
- fixed invalid documentation for #357
### non-breaking
- Added option to simplesitemap `publicBasePath`: allows the user to set the location of sitemap files hosted on the site fixes [#359]
- bumped dependencies
## 6.4.0
- added support for content_loc parsing #347 and uploader info attr
- added error handler option to sitemapstream #349 Thanks @marcoreni
## 6.3.6
- bump dependencies
## 6.3.5
- Add option to silence or redirect logs from parse #337
- `new XMLToSitemapItemStream({ logger: false })` or
- `new XMLToSitemapItemStream({ level: ErrorLevel.SILENT })` or
- `new XMLToSitemapItemStream({ logger: (level, ...message) => your.custom.logger(...message) })`
## 6.3.4
- bump dependencies
- correct return type of xmllint. Was `Promise<null>` but actually returned `Promise<void>`
- add alternate option for lang, hreflang as that is the actual name of the printed attribute
## 6.3.3
- bump ts to 4
- change file reference in sitemap-index to include .gz fixes #334
## 6.3.2
- fix unreported timing issue in SitemapAndIndexStream uncovered in latest unit tests
## 6.3.1
- fix #331 incorrect type on sourceData in simpleSitemapAndIndex.
## 6.3.0
- simpleSitemap will create the dest directory if it doesn't exist
- allow user to not gzip fixes #322
## 6.2.0
- Add simplified interface for creating sitemaps and index
- fix bug where sitemap and index stream would not properly wait to emit finish event until all sitemaps had been written
- bump deps
## 6.1.7
- Improve documentation and error messaging on ending a stream too early #317
- bump dependencies
## 6.1.6
- support allow_embed #314
- bump dependencies
## 6.1.5
- performance improvement for streamToPromise #307
## 6.1.4
- remove stale files from dist #298
- Correct documentation on renamed XMLToSitemapOptions, XMLToSitemapItemStream #297
- bump node typedef to 14.0.1
## 6.1.3
- bump node types resolves #293
## 6.1.2
- bump node types resolves #290
## 6.1.1
- Fix #286 sitemapindex tag not closing for deprecated createSitemapsAndIndex
## 6.1.0
- Added back xslUrl option removed in 5.0.0
## 6.0.0
- removed xmlbuilder as a dependency
- added stronger validity checking on values supplied to sitemap
- Added the ability to turn off or add custom xml namespaces
- CLI and library now can accept a stream which will automatically write both the index and the sitemaps. See README for usage.
### 6.0.0 breaking changes
- renamed XMLToISitemapOptions to XMLToSitemapOptions
- various error messages changed.
- removed deprecated Sitemap and SitemapIndex classes
- replaced buildSitemapIndex with SitemapIndexStream
- Typescript: various types renamed or made more specific, removed I prefix
- Typescript: view_count is now exclusively a number
- Typescript: `price:type` and `price:resolution` are now more restrictive types
- sitemap parser now returns a sitemapItem array rather than a config object that could be passed to the now removed Sitemap class
- CLI no longer accepts multiple file arguments or a mixture of file and streams except as a part of a parameter eg. prepend
## 5.1.0
Fix for #255. Baidu does not like timestamp in its sitemap.xml, this adds an option to truncate lastmod
```js
new SitemapStream({ lastmodDateOnly: true });
```
## 5.0.1
Fix for issue #254.
```sh
warning: failed to load external entity "./schema/all.xsd"
Schemas parser error : Failed to locate the main schema resource at './schema/all.xsd'.
WXS schema ./schema/all.xsd failed to compile
```
## 5.0.0
### Streams
This release is heavily focused on converting the core methods of this library to use streams. Why? Overall its made the API ~20% faster and uses only 10% or less of the memory. Some tradeoffs had to be made as in their nature streams are operate on individual segments of data as opposed to the whole. For instance, the streaming interface does not support removal of sitemap items as it does not hold on to a sitemap item after its converted to XML. It should however be possible to create your own transform that filters out entries should you desire it. The existing synchronous interfaces will remain for this release at least. Do not be surprised if they go away in a future breaking release.
### Sitemap Index
This library interface has been overhauled to use streams internally. Although it would have been preferable to convert this to a stream as well, I could not think of an interface that wouldn't actually end up more complex or confusing. It may be altered in the near future to accept a stream in addition to a simple list.
### Misc
- runnable examples, some pulled straight from README have been added to the examples directory.
- createSitemapsIndex was renamed createSitemapsAndIndex to more accurately reflect its function. It now returns a promise that resolves to true or throws with an error.
- You can now add to existing sitemap.xml files via the cli using `npx sitemap --prepend existingSitemap.xml < listOfNewURLs.json.txt`
### 5.0 Breaking Changes
- Dropped support for mobile sitemap - Google appears to have deleted their dtd and all references to it, strongly implying that they do not want you to use it. As its absence now breaks the validator, it has been dropped.
- normalizeURL(url, XMLRoot, hostname) -> normalizeURL(url, hostname)
- The second argument was unused and has been eliminated
- Support for Node 8 dropped - Node 8 is reaching its EOL December 2019
- xslURL is being dropped from all apis - styling xml is out of scope of this library.
- createSitemapIndex has been converted to a promised based api rather than callback.
- createSitemapIndex now gzips by default - pass gzip: false to disable
- cacheTime is being dropped from createSitemapIndex - This didn't actually cache the way it was written so this should be a non-breaking change in effect.
- SitemapIndex as a class has been dropped. The class did all its work on construction and there was no reason to hold on to it once you created it.
- The options for the cli have been overhauled
- `--json` is now inferred
- `--line-separated` has been flipped to `--single-line-json` to by default output options immediately compatible with feeding back into sitemap
## 4.1.1
Add a pretty print option to `toString(false)`
pass true pretty print
Add an xmlparser that will output a config that would generate that same file
cli:
use --parser to output the complete config --line-separated to print out line
separated config compatible with the --json input option for cli
lib: import parseSitemap and pass it a stream
## 4.0.2
Fix npx script error - needs the shebang
## 4.0.1
Validation functions which depend on xmllint will now warn if you do not have xmllint installed.
## 4.0.0
This release is geared around overhauling the public api for this library. Many
options have been introduced over the years and this has lead to some inconsistencies
that make the library hard to use. Most have been cleaned up but a couple notable
items remain, including the confusing names of buildSitemapIndex and createSitemapIndex
- A new experimental CLI
- stream in a list of urls stream out xml
- validate your generated sitemap
- Sitemap video item now supports id element
- Several schema errors have been cleaned up.
- Docs have been updated and streamlined.
### breaking changes
- lastmod option parses all ISO8601 date-only strings as being in UTC rather than local time
- lastmodISO is deprecated as it is equivalent to lastmod
- lastmodfile now includes the file's time as well
- lastmodrealtime is no longer necessary
- The default export of sitemap lib is now just createSitemap
- Sitemap constructor now uses a object for its constructor
```js
const { Sitemap } = require('sitemap');
const siteMap = new Sitemap({
urls = [],
hostname: 'https://example.com', // optional
cacheTime = 0,
xslUrl,
xmlNs,
level = 'warn'
})
```
- Sitemap no longer accepts a single string for its url
- Drop support for node 6
- Remove callback on toXML - This had no performance benefit
- Direct modification of urls property on Sitemap has been dropped. Use add/remove/contains
- When a Sitemap item is generated with invalid options it no longer throws by default
- instead it console warns.
- if you'd like to pre-verify your data the `validateSMIOptions` function is
now available
- To get the previous behavior pass level `createSitemap({...otheropts, level: 'throw' }) // ErrorLevel.THROW for TS users`
## 3.2.2
- revert https everywhere added in 3.2.0. xmlns is not url.
- adds alias for lastmod in the form of lastmodiso
- fixes bug in lastmod option for buildSitemapIndex where option would be overwritten if a lastmod option was provided with a single url
- fixes #201, fixes #203
## 3.2.1
- no really fixes ts errors for real this time
- fixes #193 in PR #198
## 3.2.0
- fixes #192, fixes #193 typescript errors
- correct types on player:loc and restriction:relationship types
- use https urls in xmlns
## 3.1.0
- fixes #187, #188 typescript errors
- adds support for full precision priority #176
## 3.0.0
- Converted project to typescript
- properly encode URLs #179
- updated core dependency
### 3.0 breaking changes
This will likely not break anyone's code but we're bumping to be safe
- root domain URLs are now suffixed with / (eg. `https://www.ya.ru` -> `https://www.ya.ru/`) This is a side-effect of properly encoding passed in URLs
node_modules/sitemap/CLAUDE.md
Generated Vendored
+346
View File
@@ -0,0 +1,346 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
sitemap.js is a TypeScript library and CLI tool for generating sitemap XML files compliant with the sitemaps.org protocol. It supports streaming large datasets, handles sitemap indexes for >50k URLs, and includes parsers for reading existing sitemaps.
## Development Commands
### Building
```bash
npm run build # Compile TypeScript to dist/esm/ and dist/cjs/
npm run build:esm # Build ESM only (dist/esm/)
npm run build:cjs # Build CJS only (dist/cjs/)
```
### Testing
```bash
npm test # Run Jest tests with coverage
npm run test:full # Run lint, build, Jest, and xmllint validation
npm run test:typecheck # Type check only (tsc)
npm run test:perf # Run performance tests (tests/perf.mjs)
npm run test:xmllint # Validate XML schema (requires xmllint)
```
### Linting
```bash
npx eslint lib/* ./cli.ts # Lint TypeScript files
npx eslint lib/* ./cli.ts --fix # Auto-fix linting issues
```
### Running CLI Locally
```bash
node dist/esm/cli.js < urls.txt # Run CLI from built dist
./dist/esm/cli.js --version # Run directly (has shebang)
npm link && sitemap --version # Link and test as global command
```
## Code Architecture
### Entry Points
- **[index.ts](index.ts)**: Main library entry point, exports all public APIs
- **[cli.ts](cli.ts)**: Command-line interface for generating/parsing sitemaps
### File Organization & Responsibilities
The library follows a strict separation of concerns. Each file has a specific purpose:
**Core Infrastructure:**
- **[lib/types.ts](lib/types.ts)**: ALL TypeScript type definitions, interfaces, and enums. NO implementation code.
- **[lib/constants.ts](lib/constants.ts)**: Single source of truth for all shared constants (limits, regexes, defaults).
- **[lib/validation.ts](lib/validation.ts)**: ALL validation logic, type guards, and validators centralized here.
- **[lib/utils.ts](lib/utils.ts)**: Stream utilities, URL normalization, and general helper functions.
- **[lib/errors.ts](lib/errors.ts)**: Custom error class definitions.
- **[lib/sitemap-xml.ts](lib/sitemap-xml.ts)**: Low-level XML generation utilities (text escaping, tag building).
**Stream Processing:**
- **[lib/sitemap-stream.ts](lib/sitemap-stream.ts)**: Main transform stream for URL → sitemap XML.
- **[lib/sitemap-item-stream.ts](lib/sitemap-item-stream.ts)**: Lower-level stream for sitemap item → XML elements.
- **[lib/sitemap-index-stream.ts](lib/sitemap-index-stream.ts)**: Streams for sitemap indexes and multi-file generation.
**Parsers:**
- **[lib/sitemap-parser.ts](lib/sitemap-parser.ts)**: Parses sitemap XML → SitemapItem objects.
- **[lib/sitemap-index-parser.ts](lib/sitemap-index-parser.ts)**: Parses sitemap index XML → IndexItem objects.
**High-Level API:**
- **[lib/sitemap-simple.ts](lib/sitemap-simple.ts)**: Simplified API for common use cases.
### Core Streaming Architecture
The library is built on Node.js Transform streams for memory-efficient processing of large URL lists:
**Stream Chain Flow:**
```
Input → Transform Stream → Output
```
**Key Stream Classes:**
1. **SitemapStream** ([lib/sitemap-stream.ts](lib/sitemap-stream.ts))
- Core Transform stream that converts `SitemapItemLoose` objects to sitemap XML
- Handles single sitemaps (up to ~50k URLs)
- Automatically generates XML namespaces for images, videos, news, xhtml
- Uses `SitemapItemStream` internally for XML element generation
2. **SitemapAndIndexStream** ([lib/sitemap-index-stream.ts](lib/sitemap-index-stream.ts))
- Higher-level stream for handling >50k URLs
- Automatically splits into multiple sitemap files when limit reached
- Generates sitemap index XML pointing to individual sitemaps
- Requires `getSitemapStream` callback to create output files
3. **SitemapItemStream** ([lib/sitemap-item-stream.ts](lib/sitemap-item-stream.ts))
- Low-level Transform stream that converts sitemap items to XML elements
- Validates and normalizes URLs
- Handles image, video, news, and link extensions
4. **XMLToSitemapItemStream** ([lib/sitemap-parser.ts](lib/sitemap-parser.ts))
- Parser that converts sitemap XML back to `SitemapItem` objects
- Built on SAX parser for streaming large XML files
5. **SitemapIndexStream** ([lib/sitemap-index-stream.ts](lib/sitemap-index-stream.ts))
- Generates sitemap index XML from a list of sitemap URLs
- Used for organizing multiple sitemaps
### Type System
**[lib/types.ts](lib/types.ts)** defines the core data structures:
- **SitemapItemLoose**: Flexible input type (accepts strings, objects, arrays for images/videos)
- **SitemapItem**: Strict normalized type (arrays only)
- **ErrorLevel**: Enum controlling validation behavior (SILENT, WARN, THROW)
- **NewsItem**, **Img**, **VideoItem**, **LinkItem**: Extension types for rich sitemap entries
- **IndexItem**: Structure for sitemap index entries
- **StringObj**: Generic object with string keys (used for XML attributes)
### Constants & Limits
**[lib/constants.ts](lib/constants.ts)** is the single source of truth for:
- `LIMITS`: Security limits (max URL length, max items per sitemap, max video tags, etc.)
- `DEFAULT_SITEMAP_ITEM_LIMIT`: Default items per sitemap file (45,000)
All limits are documented with references to sitemaps.org and Google specifications.
### Validation & Normalization
**[lib/validation.ts](lib/validation.ts)** centralizes ALL validation logic:
- `validateSMIOptions()`: Validates complete sitemap item fields
- `validateURL()`, `validatePath()`, `validateLimit()`: Input validation
- `validators`: Regex patterns for field validation (price, language, genres, etc.)
- Type guards: `isPriceType()`, `isResolution()`, `isValidChangeFreq()`, `isValidYesNo()`, `isAllowDeny()`
**[lib/utils.ts](lib/utils.ts)** contains utility functions:
- `normalizeURL()`: Converts `SitemapItemLoose` to `SitemapItem` with validation
- `lineSeparatedURLsToSitemapOptions()`: Stream transform for parsing line-delimited URLs
- `ReadlineStream`: Helper for reading line-by-line input
- `mergeStreams()`: Combines multiple streams into one
### XML Generation
**[lib/sitemap-xml.ts](lib/sitemap-xml.ts)** provides low-level XML building functions:
- Tag generation helpers (`otag`, `ctag`, `element`)
- Sitemap-specific element builders (images, videos, news, links)
### Error Handling
**[lib/errors.ts](lib/errors.ts)** defines custom error classes:
- `EmptyStream`, `EmptySitemap`: Stream validation errors
- `InvalidAttr`, `InvalidVideoFormat`, `InvalidNewsFormat`: Validation errors
- `XMLLintUnavailable`: External tool errors
## When Making Changes
### Where to Add New Code
- **New type or interface?** → Add to [lib/types.ts](lib/types.ts)
- **New constant or limit?** → Add to [lib/constants.ts](lib/constants.ts) (import from here everywhere)
- **New validation function or type guard?** → Add to [lib/validation.ts](lib/validation.ts)
- **New utility function?** → Add to [lib/utils.ts](lib/utils.ts)
- **New error class?** → Add to [lib/errors.ts](lib/errors.ts)
- **New public API?** → Export from [index.ts](index.ts)
### Common Pitfalls to Avoid
1. **DON'T duplicate constants** - Always import from [lib/constants.ts](lib/constants.ts)
2. **DON'T define types in implementation files** - Put them in [lib/types.ts](lib/types.ts)
3. **DON'T scatter validation logic** - Keep it all in [lib/validation.ts](lib/validation.ts)
4. **DON'T break backward compatibility** - Use re-exports if moving code between files
5. **DO update [index.ts](index.ts)** if adding new public API functions
### Adding a New Field to Sitemap Items
1. Add type to [lib/types.ts](lib/types.ts) in both `SitemapItem` and `SitemapItemLoose` interfaces
2. Add XML generation logic in [lib/sitemap-item-stream.ts](lib/sitemap-item-stream.ts) `_transform` method
3. Add parsing logic in [lib/sitemap-parser.ts](lib/sitemap-parser.ts) SAX event handlers
4. Add validation in [lib/validation.ts](lib/validation.ts) `validateSMIOptions` if needed
5. Add constants to [lib/constants.ts](lib/constants.ts) if limits are needed
6. Write tests covering the new field
### Before Submitting Changes
```bash
npm run test:full # Run all tests, linting, and validation
npm run build # Ensure both ESM and CJS builds work
npm test # Verify 90%+ code coverage maintained
```
## Finding Code in the Codebase
### "Where is...?"
- **Validation for sitemap items?** → [lib/validation.ts](lib/validation.ts) (`validateSMIOptions`)
- **URL validation?** → [lib/validation.ts](lib/validation.ts) (`validateURL`)
- **Constants like max URL length?** → [lib/constants.ts](lib/constants.ts) (`LIMITS`)
- **Type guards (isPriceType, isValidYesNo)?** → [lib/validation.ts](lib/validation.ts)
- **Type definitions (SitemapItem, etc)?** → [lib/types.ts](lib/types.ts)
- **XML escaping/generation?** → [lib/sitemap-xml.ts](lib/sitemap-xml.ts)
- **URL normalization?** → [lib/utils.ts](lib/utils.ts) (`normalizeURL`)
- **Stream utilities?** → [lib/utils.ts](lib/utils.ts) (`mergeStreams`, `lineSeparatedURLsToSitemapOptions`)
### "How do I...?"
- **Check if a value is valid?** → Import type guard from [lib/validation.ts](lib/validation.ts)
- **Get a constant limit?** → Import `LIMITS` from [lib/constants.ts](lib/constants.ts)
- **Validate user input?** → Use validation functions from [lib/validation.ts](lib/validation.ts)
- **Generate XML safely?** → Use functions from [lib/sitemap-xml.ts](lib/sitemap-xml.ts) (auto-escapes)
## Testing Strategy
Tests are in [tests/](tests/) directory with Jest:
- **[tests/sitemap-stream.test.ts](tests/sitemap-stream.test.ts)**: Core streaming functionality
- **[tests/sitemap-parser.test.ts](tests/sitemap-parser.test.ts)**: XML parsing
- **[tests/sitemap-index.test.ts](tests/sitemap-index.test.ts)**: Index generation
- **[tests/sitemap-simple.test.ts](tests/sitemap-simple.test.ts)**: High-level API
- **[tests/cli.test.ts](tests/cli.test.ts)**: CLI argument parsing
- **[tests/*-security.test.ts](tests/)**: Security-focused validation and injection tests
- **[tests/sitemap-utils.test.ts](tests/sitemap-utils.test.ts)**: Utility function tests
### Coverage Requirements (enforced by jest.config.cjs)
- Branches: 80%
- Functions: 90%
- Lines: 90%
- Statements: 90%
### When to Write Tests
- **Always** write tests for new validation functions
- **Always** write tests for new security features
- **Always** add security tests for user-facing inputs (URL validation, path traversal, etc.)
- Write tests for bug fixes to prevent regression
- Add edge case tests for data transformations
## TypeScript Configuration
The project uses a dual-build setup for ESM and CommonJS:
- **[tsconfig.json](tsconfig.json)**: ESM build (`module: "NodeNext"`, `moduleResolution: "NodeNext"`)
- Outputs to `dist/esm/`
- Includes both [index.ts](index.ts) and [cli.ts](cli.ts)
- ES2023 target with strict null checks enabled
- **[tsconfig.cjs.json](tsconfig.cjs.json)**: CommonJS build (`module: "CommonJS"`)
- Outputs to `dist/cjs/`
- Excludes [cli.ts](cli.ts) (CLI is ESM-only)
- Only includes [index.ts](index.ts) for library exports
**Important**: All relative imports must include `.js` extensions for ESM compatibility (e.g., `import { foo } from './types.js'`)
## Key Patterns
### Stream Creation
Always create a new stream instance per operation. Streams cannot be reused.
```typescript
const stream = new SitemapStream({ hostname: 'https://example.com' });
stream.write({ url: '/page' });
stream.end();
```
### Memory Management
For large datasets, use streaming patterns with `pipe()` rather than collecting all data in memory:
```typescript
// Good - streams through
lineSeparatedURLsToSitemapOptions(readStream).pipe(sitemapStream).pipe(outputStream);
// Bad - loads everything into memory
const allUrls = await readAllUrls();
allUrls.forEach(url => stream.write(url));
```
### Error Levels
Control validation strictness with `ErrorLevel`:
- `SILENT`: Skip validation (fastest, use in production if data is pre-validated)
- `WARN`: Log warnings (default, good for development)
- `THROW`: Throw on invalid data (strict mode, good for testing)
## Package Distribution
The package is distributed as a dual ESM/CommonJS package with `"type": "module"` in package.json:
- **ESM**: `dist/esm/index.js` (ES modules)
- **CJS**: `dist/cjs/index.js` (CommonJS, via conditional exports)
- **Types**: `dist/esm/index.d.ts` (TypeScript definitions)
- **Binary**: `dist/esm/cli.js` (ESM-only CLI, executable via `npx sitemap`)
- **Engines**: Node.js >=20.19.5, npm >=10.8.2
### Dual Package Exports
The `exports` field in package.json provides conditional exports:
```json
{
"exports": {
".": {
"import": "./dist/esm/index.js",
"require": "./dist/cjs/index.js"
}
}
}
```
This allows both:
```javascript
// ESM
import { SitemapStream } from 'sitemap'
// CommonJS
const { SitemapStream } = require('sitemap')
```
## Git Hooks
Husky pre-commit hooks run lint-staged which:
- Sorts package.json
- Runs eslint --fix on TypeScript files
- Runs prettier on TypeScript files
## Architecture Decisions
### Why This File Structure?
The codebase is organized around **separation of concerns** and **single source of truth** principles:
1. **Types in [lib/types.ts](lib/types.ts)**: All interfaces and enums live here, with NO implementation code. This makes types easy to find and prevents circular dependencies.
2. **Constants in [lib/constants.ts](lib/constants.ts)**: All shared constants (limits, regexes) defined once. This prevents inconsistencies where different files use different values.
3. **Validation in [lib/validation.ts](lib/validation.ts)**: All validation logic centralized. Easy to find, test, and maintain security rules.
4. **Clear file boundaries**: Each file has ONE responsibility. You know exactly where to look for specific functionality.
### Key Principles
- **Single Source of Truth**: Constants and validation logic exist in exactly one place
- **No Duplication**: Import shared code rather than copying it
- **Backward Compatibility**: Use re-exports when moving code between files to avoid breaking changes
- **Types Separate from Implementation**: [lib/types.ts](lib/types.ts) contains only type definitions
- **Security First**: All validation and limits are centralized for consistent security enforcement
### Benefits of This Organization
- **Discoverability**: Developers know exactly where to look for types, constants, or validation
- **Maintainability**: Changes to limits or validation only require editing one file
- **Consistency**: Importing from a single source prevents different parts of the code using different limits
- **Testing**: Centralized validation makes it easy to write comprehensive security tests
- **Refactoring**: Clear boundaries make it safe to refactor without affecting other modules
node_modules/sitemap/CODE_OF_CONDUCT.md
Generated Vendored
+76
View File
@@ -0,0 +1,76 @@
# Contributor Covenant Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socio-economic status, nationality, personal
appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment
include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at sitemap-cc@nimblerendition.com. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see
https://www.contributor-covenant.org/faq
node_modules/sitemap/LICENSE
Generated Vendored
+22
View File
@@ -0,0 +1,22 @@
(The MIT License)
Copyright (c) 2011 Eugene Kalinin
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
node_modules/sitemap/README.md
Generated Vendored
+355
View File
@@ -0,0 +1,355 @@
# sitemap ![MIT License](https://img.shields.io/npm/l/sitemap)[![Build Status](https://github.com/ekalinin/sitemap.js/workflows/Node%20CI/badge.svg)](https://github.com/ekalinin/sitemap.js/actions)![Monthly Downloads](https://img.shields.io/npm/dm/sitemap)
**sitemap** is a high-level streaming sitemap-generating library/CLI that
makes creating [sitemap XML](http://www.sitemaps.org/) files easy. [What is a sitemap?](https://support.google.com/webmasters/answer/156184?hl=en&ref_topic=4581190)
## Table of Contents
- [Installation](#installation)
- [Generate a one time sitemap from a list of urls](#generate-a-one-time-sitemap-from-a-list-of-urls)
- [Example of using sitemap.js with](#serve-a-sitemap-from-a-server-and-periodically-update-it) [express](https://expressjs.com/)
- [Generating more than one sitemap](#create-sitemap-and-index-files-from-one-large-list)
- [Options you can pass](#options-you-can-pass)
- [Examples](#examples)
- [API](#api)
- [Maintainers](#maintainers)
- [License](#license)
## Installation
```sh
npm install --save sitemap
```
## Generate a one time sitemap from a list of urls
If you are just looking to take a giant list of URLs and turn it into some sitemaps, try out our CLI. The cli can also parse, update and validate existing sitemaps.
```sh
npx sitemap < listofurls.txt # `npx sitemap -h` for more examples and a list of options.
```
For programmatic one time generation of a sitemap try:
```js
// ESM
import { SitemapStream, streamToPromise } from 'sitemap'
import { Readable } from 'stream'
// CommonJS
const { SitemapStream, streamToPromise } = require('sitemap')
const { Readable } = require('stream')
// An array with your links
const links = [{ url: '/page-1/', changefreq: 'daily', priority: 0.3 }]
// Create a stream to write to
const stream = new SitemapStream( { hostname: 'https://...' } )
// Return a promise that resolves with your XML string
return streamToPromise(Readable.from(links).pipe(stream)).then((data) =>
data.toString()
)
```
## Serve a sitemap from a server and periodically update it
Use this if you have less than 50 thousand urls. See SitemapAndIndexStream for if you have more.
```js
// ESM
import express from 'express'
import { SitemapStream, streamToPromise } from 'sitemap'
import { createGzip } from 'zlib'
import { Readable } from 'stream'
// CommonJS
const express = require('express')
const { SitemapStream, streamToPromise } = require('sitemap')
const { createGzip } = require('zlib')
const { Readable } = require('stream')
const app = express()
let sitemap
app.get('/sitemap.xml', function(req, res) {
res.header('Content-Type', 'application/xml');
res.header('Content-Encoding', 'gzip');
// if we have a cached entry send it
if (sitemap) {
res.send(sitemap)
return
}
try {
const smStream = new SitemapStream({ hostname: 'https://example.com/' })
const pipeline = smStream.pipe(createGzip())
// pipe your entries or directly write them.
smStream.write({ url: '/page-1/', changefreq: 'daily', priority: 0.3 })
smStream.write({ url: '/page-2/', changefreq: 'monthly', priority: 0.7 })
smStream.write({ url: '/page-3/'}) // changefreq: 'weekly', priority: 0.5
smStream.write({ url: '/page-4/', img: "http://urlTest.com" })
/* or use
Readable.from([{url: '/page-1'}...]).pipe(smStream)
if you are looking to avoid writing your own loop.
*/
// cache the response
streamToPromise(pipeline).then(sm => sitemap = sm)
// make sure to attach a write stream such as streamToPromise before ending
smStream.end()
// stream write the response
pipeline.pipe(res).on('error', (e) => {throw e})
} catch (e) {
console.error(e)
res.status(500).end()
}
})
app.listen(3000, () => {
console.log('listening')
});
```
## Create sitemap and index files from one large list
If you know you are definitely going to have more than 50,000 urls in your sitemap, you can use this slightly more complex interface to create a new sitemap every 45,000 entries and add that file to a sitemap index.
```js
// ESM
import { createReadStream, createWriteStream } from 'fs'
import { resolve } from 'path'
import { createGzip } from 'zlib'
import { simpleSitemapAndIndex, lineSeparatedURLsToSitemapOptions } from 'sitemap'
// CommonJS
const { createReadStream, createWriteStream } = require('fs')
const { resolve } = require('path')
const { createGzip } = require('zlib')
const {
simpleSitemapAndIndex,
lineSeparatedURLsToSitemapOptions
} = require('sitemap')
// writes sitemaps and index out to the destination you provide.
simpleSitemapAndIndex({
hostname: 'https://example.com',
destinationDir: './',
sourceData: lineSeparatedURLsToSitemapOptions(
createReadStream('./your-data.json.txt')
),
// sourceData can also be:
// sourceData: [{ url: '/page-1/', changefreq: 'daily'}, ...],
// or
// sourceData: './your-data.json.txt',
limit: 45000, // optional, default: 50000
gzip: true, // optional, default: true
publicBasePath: '/sitemaps/', // optional, default: './'
xslUrl: 'https://example.com/sitemap.xsl', // optional XSL stylesheet
}).then(() => {
// Do follow up actions
})
```
Want to customize that?
```js
// ESM
import { createReadStream, createWriteStream } from 'fs'
import { resolve } from 'path'
import { createGzip } from 'zlib'
import { Readable } from 'stream'
import { SitemapAndIndexStream, SitemapStream, lineSeparatedURLsToSitemapOptions } from 'sitemap'
// CommonJS
const { createReadStream, createWriteStream } = require('fs')
const { resolve } = require('path')
const { createGzip } = require('zlib')
const { Readable } = require('stream')
const {
SitemapAndIndexStream,
SitemapStream,
lineSeparatedURLsToSitemapOptions
} = require('sitemap')
const sms = new SitemapAndIndexStream({
limit: 50000, // defaults to 45k
lastmodDateOnly: false, // print date not time
// SitemapAndIndexStream will call this user provided function every time
// it needs to create a new sitemap file. You merely need to return a stream
// for it to write the sitemap urls to and the expected url where that sitemap will be hosted
getSitemapStream: (i) => {
const sitemapStream = new SitemapStream({ hostname: 'https://example.com' });
// if your server automatically serves sitemap.xml.gz when requesting sitemap.xml leave this line be
// otherwise you will need to add .gz here and remove it a couple lines below so that both the index
// and the actual file have a .gz extension
const path = `./sitemap-${i}.xml`;
const ws = sitemapStream
.pipe(createGzip()) // compress the output of the sitemap
.pipe(createWriteStream(resolve(path + '.gz'))); // write it to sitemap-NUMBER.xml
return [new URL(path, 'https://example.com/subdir/').toString(), sitemapStream, ws];
},
});
// when reading from a file
lineSeparatedURLsToSitemapOptions(
createReadStream('./your-data.json.txt')
)
.pipe(sms)
.pipe(createGzip())
.pipe(createWriteStream(resolve('./sitemap-index.xml.gz')));
// or reading straight from an in-memory array
sms
.pipe(createGzip())
.pipe(createWriteStream(resolve('./sitemap-index.xml.gz')));
const arrayOfSitemapItems = [{ url: '/page-1/', changefreq: 'daily'}, ...]
Readable.from(arrayOfSitemapItems).pipe(sms)
// or
arrayOfSitemapItems.forEach(item => sms.write(item))
sms.end() // necessary to let it know you've got nothing else to write
```
### Options you can pass
```js
// ESM
import { SitemapStream, streamToPromise } from 'sitemap'
// CommonJS
const { SitemapStream, streamToPromise } = require('sitemap')
const smStream = new SitemapStream({
hostname: 'http://www.mywebsite.com',
xslUrl: "https://example.com/style.xsl",
lastmodDateOnly: false, // print date not time
xmlns: { // trim the xml namespace
news: true, // flip to false to omit the xml namespace for news
xhtml: true,
image: true,
video: true,
custom: [
'xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"',
'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"',
],
}
})
// coalesce stream to value
// alternatively you can pipe to another stream
streamToPromise(smStream).then(console.log)
smStream.write({
url: '/page1',
changefreq: 'weekly',
priority: 0.8, // A hint to the crawler that it should prioritize this over items less than 0.8
})
// each sitemap entry supports many options
// See [Sitemap Item Options](./api.md#sitemap-item-options) below for details
smStream.write({
url: 'http://test.com/page-1/',
img: [
{
url: 'http://test.com/img1.jpg',
caption: 'An image',
title: 'The Title of Image One',
geoLocation: 'London, United Kingdom',
license: 'https://creativecommons.org/licenses/by/4.0/'
},
{
url: 'http://test.com/img2.jpg',
caption: 'Another image',
title: 'The Title of Image Two',
geoLocation: 'London, United Kingdom',
license: 'https://creativecommons.org/licenses/by/4.0/'
}
],
video: [
{
thumbnail_loc: 'http://test.com/tmbn1.jpg',
title: 'A video title',
description: 'This is a video'
},
{
thumbnail_loc: 'http://test.com/tmbn2.jpg',
title: 'A video with an attribute',
description: 'This is another video',
'player_loc': 'http://www.example.com/videoplayer.mp4?video=123',
'player_loc:autoplay': 'ap=1',
'player_loc:allow_embed': 'yes'
}
],
links: [
{ lang: 'en', url: 'http://test.com/page-1/' },
{ lang: 'ja', url: 'http://test.com/page-1/ja/' }
],
androidLink: 'android-app://com.company.test/page-1/',
news: {
publication: {
name: 'The Example Times',
language: 'en'
},
genres: 'PressRelease, Blog',
publication_date: '2008-12-23',
title: 'Companies A, B in Merger Talks',
keywords: 'business, merger, acquisition, A, B',
stock_tickers: 'NASDAQ:A, NASDAQ:B'
}
})
// indicate there is nothing left to write
smStream.end()
```
## Filtering sitemap entries during parsing
You can filter or delete items from a sitemap while parsing by piping through a custom Transform stream. This is useful when you want to selectively process only certain URLs from an existing sitemap.
```js
import { createReadStream } from 'fs'
import { Transform } from 'stream'
import { XMLToSitemapItemStream } from 'sitemap'
// Create a filter that only keeps certain URLs
const filterStream = new Transform({
objectMode: true,
transform(item, encoding, callback) {
// Only keep URLs containing '/blog/'
if (item.url.includes('/blog/')) {
callback(undefined, item) // Keep this item
} else {
callback() // Skip this item (effectively "deleting" it)
}
}
})
// Parse and filter
createReadStream('./sitemap.xml')
.pipe(new XMLToSitemapItemStream())
.pipe(filterStream)
.on('data', (item) => {
console.log('Filtered URL:', item.url)
})
```
You can also chain multiple filters together, filter based on priority/changefreq, or use the filtered results to generate a new sitemap. See [examples/filter-sitemap.js](./examples/filter-sitemap.js) for more filtering patterns.
## Examples
For more examples see the [examples directory](./examples/)
## API
Full API docs can be found [here](./api.md)
## Maintainers
- [@ekalinin](https://github.com/ekalinin)
- [@derduher](https://github.com/derduher)
## License
See [LICENSE](https://github.com/ekalinin/sitemap.js/blob/master/LICENSE) file.
node_modules/sitemap/api.md
Generated Vendored
+454
View File
@@ -0,0 +1,454 @@
# API
- [API](#api)
- [SitemapStream](#sitemapstream)
- [XMLToSitemapItemStream](#xmltositemapitemstream)
- [SitemapAndIndexStream](#sitemapandindexstream)
- [simpleSitemapAndIndex](#simplesitemapandindex)
- [SitemapIndexStream](#sitemapindexstream)
- [xmlLint](#xmllint)
- [parseSitemap](#parsesitemap)
- [lineSeparatedURLsToSitemapOptions](#lineseparatedurlstositemapoptions)
- [streamToPromise](#streamtopromise)
- [ObjectStreamToJSON](#objectstreamtojson)
- [SitemapItemStream](#sitemapitemstream)
- [Sitemap Item Options](#sitemap-item-options)
- [SitemapImage](#sitemapimage)
- [VideoItem](#videoitem)
- [LinkItem](#linkitem)
- [NewsItem](#newsitem)
## SitemapStream
A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream) for turning a [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams) of either [SitemapItemOptions](#sitemap-item-options) or url strings into a Sitemap. The readable stream it transforms **must** be in object mode.
```javascript
// ESM
import { SitemapStream } from 'sitemap'
// CommonJS
const { SitemapStream } = require('sitemap')
const sms = new SitemapStream({
hostname: 'https://example.com', // optional only necessary if your paths are relative
lastmodDateOnly: false // defaults to false, flip to true for baidu
xmlns: { // XML namespaces to turn on - all by default
news: true,
xhtml: true,
image: true,
video: true,
// custom: ['xmlns:custom="https://example.com"']
},
errorHandler: undefined // defaults to a standard errorLogger that logs to console or throws if the errorLevel is set to throw
})
const readable = // a readable stream of objects
readable.pipe(sms).pipe(process.stdout)
```
## XMLToSitemapItemStream
Takes a stream of xml and transforms it into a stream of SitemapOptions.
Use this to parse existing sitemaps into config options compatible with this library
```javascript
// ESM
import { createReadStream, createWriteStream } from 'fs';
import { XMLToSitemapItemStream, ObjectStreamToJSON, ErrorLevel } from 'sitemap';
// CommonJS
const { createReadStream, createWriteStream } = require('fs');
const { XMLToSitemapItemStream, ObjectStreamToJSON, ErrorLevel } = require('sitemap');
createReadStream('./some/sitemap.xml')
// turn the xml into sitemap option item options
.pipe(new XMLToSitemapItemStream({
// optional
level: ErrorLevel.WARN // default is WARN pass SILENT to silence
logger: false // default is console log, pass false as another way to silence or your own custom logger
}))
// convert the object stream to JSON
.pipe(new ObjectStreamToJSON())
// write the library compatible options to disk
.pipe(createWriteStream('./sitemapOptions.json'))
```
## SitemapAndIndexStream
Use this to take a stream which may go over the max of 50000 items and split it into an index and sitemaps.
SitemapAndIndexStream consumes a stream of urls and streams out index entries while writing individual urls to the streams you give it.
Provide it with a function which when provided with a index returns a url where the sitemap will ultimately be hosted and a stream to write the current sitemap to. This function will be called everytime the next item in the stream would exceed the provided limit.
```js
// ESM
import { createReadStream, createWriteStream } from 'fs';
import { resolve } from 'path';
import { createGzip } from 'zlib';
import {
SitemapAndIndexStream,
SitemapStream,
lineSeparatedURLsToSitemapOptions
} from 'sitemap';
// CommonJS
const { createReadStream, createWriteStream } = require('fs');
const { resolve } = require('path');
const { createGzip } = require('zlib');
const {
SitemapAndIndexStream,
SitemapStream,
lineSeparatedURLsToSitemapOptions
} = require('sitemap');
const sms = new SitemapAndIndexStream({
limit: 10000, // defaults to 45k
// SitemapAndIndexStream will call this user provided function every time
// it needs to create a new sitemap file. You merely need to return a stream
// for it to write the sitemap urls to and the expected url where that sitemap will be hosted
getSitemapStream: (i) => {
const sitemapStream = new SitemapStream();
const path = `./sitemap-${i}.xml`;
const ws = sitemapStream
.pipe(createGzip()) // compress the output of the sitemap
.pipe(createWriteStream(resolve(path + '.gz'))); // write it to sitemap-NUMBER.xml
return [new URL(path, 'https://example.com/subdir/').toString(), sitemapStream, ws];
},
});
lineSeparatedURLsToSitemapOptions(
createReadStream('./your-data.json.txt')
)
.pipe(sms)
.pipe(createGzip())
.pipe(createWriteStream(resolve('./sitemap-index.xml.gz')));
```
## simpleSitemapAndIndex
A simpler interface for creating sitemaps and indexes. Automatically handles splitting large datasets into multiple sitemap files.
```js
// ESM
import { simpleSitemapAndIndex } from 'sitemap';
// CommonJS
const { simpleSitemapAndIndex } = require('sitemap');
// writes sitemaps and index out to the destination you provide.
await simpleSitemapAndIndex({
hostname: 'https://example.com',
destinationDir: './',
sourceData: [
{ url: '/page-1/', changefreq: 'daily', priority: 0.3 },
{ url: '/page-2/', changefreq: 'weekly', priority: 0.7 },
// ... more URLs
],
// optional: limit URLs per sitemap (default: 50000, must be 1-50000)
limit: 45000,
// optional: gzip the output files (default: true)
gzip: true,
// optional: public base path for sitemap URLs (default: './')
publicBasePath: '/sitemaps/',
// optional: XSL stylesheet URL for XML display
xslUrl: 'https://example.com/sitemap.xsl',
// or read from a file
// sourceData: lineSeparatedURLsToSitemapOptions(createReadStream('./urls.txt')),
// or
// sourceData: './urls.txt',
});
```
### Options
- **hostname** (required): The base URL for all sitemap entries. Must be a valid `http://` or `https://` URL.
- **sitemapHostname** (optional): The base URL for sitemap index entries if different from `hostname`. Must be a valid `http://` or `https://` URL.
- **destinationDir** (required): Directory where sitemaps and index will be written. Can be relative or absolute, but must not contain path traversal sequences (`..`).
- **sourceData** (required): URL source data. Can be:
- Array of strings (URLs)
- Array of `SitemapItemLoose` objects
- String (file path to line-separated URLs)
- Readable stream
- **limit** (optional): Maximum URLs per sitemap file. Must be between 1 and 50,000 per [sitemaps.org spec](https://www.sitemaps.org/protocol.html). Default: 50000
- **gzip** (optional): Whether to gzip compress the output files. Default: true
- **publicBasePath** (optional): Base path for sitemap URLs in the index. Must not contain path traversal sequences. Default: './'
- **xslUrl** (optional): URL to an XSL stylesheet for XML display. Must be a valid `http://` or `https://` URL.
### Security
`simpleSitemapAndIndex` includes comprehensive security validation to protect against common attacks:
**URL Validation:**
- All URLs (hostname, sitemapHostname) must use `http://` or `https://` protocols only
- Maximum URL length enforced at 2048 characters per sitemaps.org specification
- URLs are parsed and validated to ensure they are well-formed
**Path Traversal Protection:**
- `destinationDir` and `publicBasePath` are checked for path traversal sequences (`..`)
- Validation detects `..` in all positions (beginning, middle, end, standalone)
- Both Unix-style (`/`) and Windows-style (`\`) path separators are normalized and checked
- Null bytes (`\0`) are rejected to prevent path manipulation attacks
**XSL Stylesheet Security:**
- XSL URLs must use `http://` or `https://` protocols
- Case-insensitive checks block dangerous content patterns:
- Script tags: `<script`, `<ScRiPt`, `<SCRIPT>`, etc.
- Dangerous protocols: `javascript:`, `data:`, `vbscript:`, `file:`, `about:`
- URL-encoded attacks: `%3cscript`, `javascript%3a`, etc.
- Maximum URL length enforced at 2048 characters
**Resource Limits:**
- Limit validated to be an integer between 1 and 50,000 per sitemaps.org specification
- Prevents resource exhaustion attacks and ensures search engine compatibility
**Data Validation:**
- Video ratings are validated to be valid numbers between 0 and 5
- Video view counts are validated to be non-negative integers
- Date values (lastmod, lastmodISO) are validated to be parseable dates
### Errors
May throw:
- `InvalidHostnameError`: Invalid or malformed hostname/sitemapHostname
- `InvalidPathError`: destinationDir contains path traversal or invalid characters
- `InvalidPublicBasePathError`: publicBasePath contains path traversal or invalid characters
- `InvalidLimitError`: limit is out of range (not 1-50,000)
- `InvalidXSLUrlError`: xslUrl is invalid or potentially malicious
- `Error`: Invalid sourceData type or file system errors
## SitemapIndexStream
Writes a sitemap index when given a stream urls.
```js
// ESM
import { SitemapIndexStream } from 'sitemap';
// CommonJS
const { SitemapIndexStream } = require('sitemap');
/**
* writes the following
* <?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/</loc>
</sitemap>
<sitemap>
<loc>https://example.com/2</loc>
</sitemap>
*/
const smis = new SitemapIndexStream({level: 'warn'})
smis.write({url: 'https://example.com/'})
smis.write({url: 'https://example.com/2'})
smis.pipe(writestream)
smis.end()
```
## xmlLint
Resolve or reject depending on whether the passed in xml is a valid sitemap.
This is just a wrapper around the xmlLint command line tool and thus requires
xmlLint to be installed on the system.
**Security Note:** This function accepts XML content as a string or Readable stream
and always pipes it via stdin to xmllint. It does NOT accept file paths to prevent
command injection vulnerabilities.
```js
// ESM
import { createReadStream, readFileSync } from 'fs';
import { xmlLint } from 'sitemap';
// CommonJS
const { createReadStream, readFileSync } = require('fs');
const { xmlLint } = require('sitemap');
// Validate using a stream
xmlLint(createReadStream('./example.xml')).then(
() => console.log('xml is valid'),
([err, stderr]) => console.error('xml is invalid', stderr)
)
// Validate using a string
const xmlContent = readFileSync('./example.xml', 'utf8');
xmlLint(xmlContent).then(
() => console.log('xml is valid'),
([err, stderr]) => console.error('xml is invalid', stderr)
)
```
## parseSitemap
Read xml and resolve with an array of sitemap items or reject with an error
```js
// ESM
import { createReadStream } from 'fs';
import { parseSitemap } from 'sitemap';
// CommonJS
const { createReadStream } = require('fs');
const { parseSitemap } = require('sitemap');
parseSitemap(createReadStream('./example.xml')).then(
(items) => {
// items is an array of sitemap items
console.log(items);
},
(err) => console.log(err)
)
```
## lineSeparatedURLsToSitemapOptions
Takes a stream of urls or sitemapoptions likely from fs.createReadStream('./path') and returns an object stream of sitemap items.
## streamToPromise
Takes a stream returns a promise that resolves when stream emits finish.
```javascript
// ESM
import { streamToPromise, SitemapStream } from 'sitemap';
// CommonJS
const { streamToPromise, SitemapStream } = require('sitemap');
const sitemap = new SitemapStream({ hostname: 'http://example.com' });
sitemap.write({ url: '/page-1/', changefreq: 'daily', priority: 0.3 })
sitemap.end()
streamToPromise(sitemap).then(buffer => console.log(buffer.toString())) // emits the full sitemap
```
## ObjectStreamToJSON
A Transform that converts a stream of objects into a JSON Array or a line separated stringified JSON.
- @param [lineSeparated=false] whether to separate entries by a new line or comma
```javascript
// ESM
import { Readable } from 'stream';
import { ObjectStreamToJSON } from 'sitemap';
// CommonJS
const { Readable } = require('stream');
const { ObjectStreamToJSON } = require('sitemap');
const stream = Readable.from([{a: 'b'}])
.pipe(new ObjectStreamToJSON())
.pipe(process.stdout)
stream.end()
// prints {"a":"b"}
```
## SitemapItemStream
Takes a stream of SitemapItemOptions and spits out xml for each
```js
// ESM
import { SitemapItemStream } from 'sitemap';
// CommonJS
const { SitemapItemStream } = require('sitemap');
// writes <url><loc>https://example.com</loc><url><url><loc>https://example.com/2</loc><url>
const smis = new SitemapItemStream({level: 'warn'})
smis.pipe(writestream)
smis.write({url: 'https://example.com', img: [], video: [], links: []})
smis.write({url: 'https://example.com/2', img: [], video: [], links: []})
smis.end()
```
## Sitemap Item Options
|Option|Type|eg|Description|
|------|----|--|-----------|
|url|string|`http://example.com/some/path`|The only required property for every sitemap entry|
|lastmod|string|'2019-07-29' or '2019-07-22T05:58:37.037Z'|When the page we as last modified use the W3C Datetime ISO8601 subset <https://www.sitemaps.org/protocol.html#xmlTagDefinitions>|
|changefreq|string|'weekly'|How frequently the page is likely to change. This value provides general information to search engines and may not correlate exactly to how often they crawl the page. Please note that the value of this tag is considered a hint and not a command. See <https://www.sitemaps.org/protocol.html#xmlTagDefinitions> for the acceptable values|
|priority|number|0.6|The priority of this URL relative to other URLs on your site. Valid values range from 0.0 to 1.0. This value does not affect how your pages are compared to pages on other sites—it only lets the search engines know which pages you deem most important for the crawlers. The default priority of a page is 0.5. <https://www.sitemaps.org/protocol.html#xmlTagDefinitions>|
|img|object[]|see [#SitemapImage](#sitemapimage)|<https://support.google.com/webmasters/answer/178636?hl=en&ref_topic=4581190>|
|video|object[]|see [#VideoItem](#videoitem)|<https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190>|
|links|object[]|see [#LinkItem](#linkitem)|Tell search engines about localized versions <https://support.google.com/webmasters/answer/189077>|
|news|object|see [#NewsItem](#newsitem)|<https://support.google.com/webmasters/answer/74288?hl=en&ref_topic=4581190>|
|ampLink|string|`http://ampproject.org/article.amp.html`||
|cdata|boolean|true|wrap url in cdata xml escape|
## SitemapImage
Sitemap image
<https://support.google.com/webmasters/answer/178636?hl=en&ref_topic=4581190>
|Option|Type|eg|Description|
|------|----|--|-----------|
|url|string|`http://example.com/image.jpg`|The URL of the image.|
|caption|string - optional|'Here we did the stuff'|The caption of the image.|
|title|string - optional|'Star Wars EP IV'|The title of the image.|
|geoLocation|string - optional|'Limerick, Ireland'|The geographic location of the image.|
|license|string - optional|`http://example.com/license.txt`|A URL to the license of the image.|
## VideoItem
Sitemap video. <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190>
|Option|Type|eg|Description|
|------|----|--|-----------|
|thumbnail_loc|string|`"https://rtv3-img-roosterteeth.akamaized.net/store/0e841100-289b-4184-ae30-b6a16736960a.jpg/sm/thumb3.jpg"`|A URL pointing to the video thumbnail image file|
|title|string|'2018:E6 - GoldenEye: Source'|The title of the video. |
|description|string|'We play gun game in GoldenEye: Source with a good friend of ours. His name is Gruchy. Dan Gruchy.'|A description of the video. Maximum 2048 characters. |
|content_loc|string - optional|`"http://streamserver.example.com/video123.mp4"`|A URL pointing to the actual video media file. Should be one of the supported formats. HTML is not a supported format. Flash is allowed, but no longer supported on most mobile platforms, and so may be indexed less well. Must not be the same as the `<loc>` URL.|
|player_loc|string - optional|`"https://roosterteeth.com/embed/rouletsplay-2018-goldeneye-source"`|A URL pointing to a player for a specific video. Usually this is the information in the src element of an `<embed>` tag. Must not be the same as the `<loc>` URL|
|'player_loc:autoplay'|string - optional|'ap=1'|a string the search engine can append as a query param to enable automatic playback|
|'player_loc:allow_embed'|boolean - optional|'yes'|Whether the search engine can embed the video in search results. Allowed values are yes or no.|
|duration|number - optional| 600| duration of video in seconds|
|expiration_date| string - optional|"2012-07-16T19:20:30+08:00"|The date after which the video will no longer be available|
|view_count|number - optional|'21000000000'|The number of times the video has been viewed.|
|publication_date| string - optional|"2018-04-27T17:00:00.000Z"|The date the video was first published, in W3C format.|
|category|string - optional|"Baking"|A short description of the broad category that the video belongs to. This is a string no longer than 256 characters.|
|restriction|string - optional|"IE GB US CA"|Whether to show or hide your video in search results from specific countries.|
|restriction:relationship| string - optional|"deny"||
|gallery_loc| string - optional|`"https://roosterteeth.com/series/awhu"`|Currently not used.|
|gallery_loc:title|string - optional|"awhu series page"|Currently not used.|
|price|string - optional|"1.99"|The price to download or view the video. Omit this tag for free videos.|
|price:resolution|string - optional|"HD"|Specifies the resolution of the purchased version. Supported values are hd and sd.|
|price:currency| string - optional|"USD"|currency [Required] Specifies the currency in ISO 4217 format.|
|price:type|string - optional|"rent"|type [Optional] Specifies the purchase option. Supported values are rent and own. |
|uploader|string - optional|"GrillyMcGrillerson"|The video uploader's name. Only one <video:uploader> is allowed per video. String value, max 255 characters.|
|uploader:info|string - optional|"https://example.com/about"|Specifies the URL of a webpage with additional information about this uploader. This URL must be in the same domain as the `<loc>` tag.
|platform|string - optional|"tv"|Whether to show or hide your video in search results on specified platform types. This is a list of space-delimited platform types. See <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190> for more detail|
|platform:relationship|string 'Allow'\|'Deny' - optional|'Allow'||
|id|string - optional|||
|tag|string[] - optional|['Baking']|An arbitrary string tag describing the video. Tags are generally very short descriptions of key concepts associated with a video or piece of content.|
|rating|number - optional|2.5|The rating of the video. Supported values are float numbers|
|family_friendly|string 'YES'\|'NO' - optional|'YES'||
|requires_subscription|string 'YES'\|'NO' - optional|'YES'|Indicates whether a subscription (either paid or free) is required to view the video. Allowed values are yes or no.|
|live|string 'YES'\|'NO' - optional|'NO'|Indicates whether the video is a live stream. Supported values are yes or no.|
## LinkItem
<https://support.google.com/webmasters/answer/189077>
|Option|Type|eg|Description|
|------|----|--|-----------|
|lang|string|'en'||
|url|string|`'http://example.com/en/'`||
## NewsItem
<https://support.google.com/webmasters/answer/74288?hl=en&ref_topic=4581190>
|Option|Type|eg|Description|
|------|----|--|-----------|
|access|string - 'Registration' \| 'Subscription'| 'Registration' - optional||
|publication| object|see following options||
|publication['name']| string|'The Example Times'|The `<name>` is the name of the news publication. It must exactly match the name as it appears on your articles on news.google.com, except for anything in parentheses.|
|publication['language']|string|'en'|The `<language>` is the language of your publication. Use an ISO 639 language code (2 or 3 letters).|
|genres|string - optional|'PressRelease, Blog'||
|publication_date|string|'2008-12-23'|Article publication date in W3C format, using either the "complete date" (YYYY-MM-DD) format or the "complete date plus hours, minutes, and seconds"|
|title|string|'Companies A, B in Merger Talks'|The title of the news article.|
|keywords|string - optional|"business, merger, acquisition, A, B"||
|stock_tickers|string - optional|"NASDAQ:A, NASDAQ:B"||
node_modules/sitemap/dist/cjs/index.d.ts
Generated Vendored
+17
View File
@@ -0,0 +1,17 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
export { SitemapItemStream, SitemapItemStreamOptions, } from './lib/sitemap-item-stream.js';
export { IndexTagNames, SitemapIndexStream, SitemapIndexStreamOptions, SitemapAndIndexStream, SitemapAndIndexStreamOptions, } from './lib/sitemap-index-stream.js';
export { streamToPromise, SitemapStream, SitemapStreamOptions, } from './lib/sitemap-stream.js';
export * from './lib/errors.js';
export * from './lib/types.js';
export { lineSeparatedURLsToSitemapOptions, mergeStreams, validateSMIOptions, normalizeURL, ReadlineStream, ReadlineStreamOptions, } from './lib/utils.js';
export { xmlLint } from './lib/xmllint.js';
export { parseSitemap, XMLToSitemapItemStream, XMLToSitemapItemStreamOptions, ObjectStreamToJSON, ObjectStreamToJSONOptions, } from './lib/sitemap-parser.js';
export { parseSitemapIndex, XMLToSitemapIndexStream, XMLToSitemapIndexItemStreamOptions, IndexObjectStreamToJSON, IndexObjectStreamToJSONOptions, } from './lib/sitemap-index-parser.js';
export { simpleSitemapAndIndex, SimpleSitemapAndIndexOptions, } from './lib/sitemap-simple.js';
export { validateURL, validatePath, validateLimit, validatePublicBasePath, validateXSLUrl, validators, isPriceType, isResolution, isValidChangeFreq, isValidYesNo, isAllowDeny, } from './lib/validation.js';
export { LIMITS, DEFAULT_SITEMAP_ITEM_LIMIT } from './lib/constants.js';
node_modules/sitemap/dist/cjs/index.js
Generated Vendored
+66
View File
@@ -0,0 +1,66 @@
"use strict";
var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
if (k2 === undefined) k2 = k;
var desc = Object.getOwnPropertyDescriptor(m, k);
if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
desc = { enumerable: true, get: function() { return m[k]; } };
}
Object.defineProperty(o, k2, desc);
}) : (function(o, m, k, k2) {
if (k2 === undefined) k2 = k;
o[k2] = m[k];
}));
var __exportStar = (this && this.__exportStar) || function(m, exports) {
for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
};
Object.defineProperty(exports, "__esModule", { value: true });
exports.DEFAULT_SITEMAP_ITEM_LIMIT = exports.LIMITS = exports.isAllowDeny = exports.isValidYesNo = exports.isValidChangeFreq = exports.isResolution = exports.isPriceType = exports.validators = exports.validateXSLUrl = exports.validatePublicBasePath = exports.validateLimit = exports.validatePath = exports.validateURL = exports.simpleSitemapAndIndex = exports.IndexObjectStreamToJSON = exports.XMLToSitemapIndexStream = exports.parseSitemapIndex = exports.ObjectStreamToJSON = exports.XMLToSitemapItemStream = exports.parseSitemap = exports.xmlLint = exports.ReadlineStream = exports.normalizeURL = exports.validateSMIOptions = exports.mergeStreams = exports.lineSeparatedURLsToSitemapOptions = exports.SitemapStream = exports.streamToPromise = exports.SitemapAndIndexStream = exports.SitemapIndexStream = exports.IndexTagNames = exports.SitemapItemStream = void 0;
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
var sitemap_item_stream_js_1 = require("./lib/sitemap-item-stream.js");
Object.defineProperty(exports, "SitemapItemStream", { enumerable: true, get: function () { return sitemap_item_stream_js_1.SitemapItemStream; } });
var sitemap_index_stream_js_1 = require("./lib/sitemap-index-stream.js");
Object.defineProperty(exports, "IndexTagNames", { enumerable: true, get: function () { return sitemap_index_stream_js_1.IndexTagNames; } });
Object.defineProperty(exports, "SitemapIndexStream", { enumerable: true, get: function () { return sitemap_index_stream_js_1.SitemapIndexStream; } });
Object.defineProperty(exports, "SitemapAndIndexStream", { enumerable: true, get: function () { return sitemap_index_stream_js_1.SitemapAndIndexStream; } });
var sitemap_stream_js_1 = require("./lib/sitemap-stream.js");
Object.defineProperty(exports, "streamToPromise", { enumerable: true, get: function () { return sitemap_stream_js_1.streamToPromise; } });
Object.defineProperty(exports, "SitemapStream", { enumerable: true, get: function () { return sitemap_stream_js_1.SitemapStream; } });
__exportStar(require("./lib/errors.js"), exports);
__exportStar(require("./lib/types.js"), exports);
var utils_js_1 = require("./lib/utils.js");
Object.defineProperty(exports, "lineSeparatedURLsToSitemapOptions", { enumerable: true, get: function () { return utils_js_1.lineSeparatedURLsToSitemapOptions; } });
Object.defineProperty(exports, "mergeStreams", { enumerable: true, get: function () { return utils_js_1.mergeStreams; } });
Object.defineProperty(exports, "validateSMIOptions", { enumerable: true, get: function () { return utils_js_1.validateSMIOptions; } });
Object.defineProperty(exports, "normalizeURL", { enumerable: true, get: function () { return utils_js_1.normalizeURL; } });
Object.defineProperty(exports, "ReadlineStream", { enumerable: true, get: function () { return utils_js_1.ReadlineStream; } });
var xmllint_js_1 = require("./lib/xmllint.js");
Object.defineProperty(exports, "xmlLint", { enumerable: true, get: function () { return xmllint_js_1.xmlLint; } });
var sitemap_parser_js_1 = require("./lib/sitemap-parser.js");
Object.defineProperty(exports, "parseSitemap", { enumerable: true, get: function () { return sitemap_parser_js_1.parseSitemap; } });
Object.defineProperty(exports, "XMLToSitemapItemStream", { enumerable: true, get: function () { return sitemap_parser_js_1.XMLToSitemapItemStream; } });
Object.defineProperty(exports, "ObjectStreamToJSON", { enumerable: true, get: function () { return sitemap_parser_js_1.ObjectStreamToJSON; } });
var sitemap_index_parser_js_1 = require("./lib/sitemap-index-parser.js");
Object.defineProperty(exports, "parseSitemapIndex", { enumerable: true, get: function () { return sitemap_index_parser_js_1.parseSitemapIndex; } });
Object.defineProperty(exports, "XMLToSitemapIndexStream", { enumerable: true, get: function () { return sitemap_index_parser_js_1.XMLToSitemapIndexStream; } });
Object.defineProperty(exports, "IndexObjectStreamToJSON", { enumerable: true, get: function () { return sitemap_index_parser_js_1.IndexObjectStreamToJSON; } });
var sitemap_simple_js_1 = require("./lib/sitemap-simple.js");
Object.defineProperty(exports, "simpleSitemapAndIndex", { enumerable: true, get: function () { return sitemap_simple_js_1.simpleSitemapAndIndex; } });
var validation_js_1 = require("./lib/validation.js");
Object.defineProperty(exports, "validateURL", { enumerable: true, get: function () { return validation_js_1.validateURL; } });
Object.defineProperty(exports, "validatePath", { enumerable: true, get: function () { return validation_js_1.validatePath; } });
Object.defineProperty(exports, "validateLimit", { enumerable: true, get: function () { return validation_js_1.validateLimit; } });
Object.defineProperty(exports, "validatePublicBasePath", { enumerable: true, get: function () { return validation_js_1.validatePublicBasePath; } });
Object.defineProperty(exports, "validateXSLUrl", { enumerable: true, get: function () { return validation_js_1.validateXSLUrl; } });
Object.defineProperty(exports, "validators", { enumerable: true, get: function () { return validation_js_1.validators; } });
Object.defineProperty(exports, "isPriceType", { enumerable: true, get: function () { return validation_js_1.isPriceType; } });
Object.defineProperty(exports, "isResolution", { enumerable: true, get: function () { return validation_js_1.isResolution; } });
Object.defineProperty(exports, "isValidChangeFreq", { enumerable: true, get: function () { return validation_js_1.isValidChangeFreq; } });
Object.defineProperty(exports, "isValidYesNo", { enumerable: true, get: function () { return validation_js_1.isValidYesNo; } });
Object.defineProperty(exports, "isAllowDeny", { enumerable: true, get: function () { return validation_js_1.isAllowDeny; } });
var constants_js_1 = require("./lib/constants.js");
Object.defineProperty(exports, "LIMITS", { enumerable: true, get: function () { return constants_js_1.LIMITS; } });
Object.defineProperty(exports, "DEFAULT_SITEMAP_ITEM_LIMIT", { enumerable: true, get: function () { return constants_js_1.DEFAULT_SITEMAP_ITEM_LIMIT; } });
node_modules/sitemap/dist/cjs/lib/constants.d.ts
Generated Vendored
+49
View File
@@ -0,0 +1,49 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
/**
* Shared constants used across the sitemap library
* This file serves as a single source of truth for limits and validation patterns
*/
/**
* Security limits for sitemap generation and parsing
*
* These limits are based on:
* - sitemaps.org protocol specification
* - Security best practices to prevent DoS and injection attacks
* - Google's sitemap extension specifications
*
* @see https://www.sitemaps.org/protocol.html
* @see https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap
*/
export declare const LIMITS: {
readonly MAX_URL_LENGTH: 2048;
readonly URL_PROTOCOL_REGEX: RegExp;
readonly MIN_SITEMAP_ITEM_LIMIT: 1;
readonly MAX_SITEMAP_ITEM_LIMIT: 50000;
readonly MAX_VIDEO_TITLE_LENGTH: 100;
readonly MAX_VIDEO_DESCRIPTION_LENGTH: 2048;
readonly MAX_VIDEO_CATEGORY_LENGTH: 256;
readonly MAX_TAGS_PER_VIDEO: 32;
readonly MAX_NEWS_TITLE_LENGTH: 200;
readonly MAX_NEWS_NAME_LENGTH: 256;
readonly MAX_IMAGE_CAPTION_LENGTH: 512;
readonly MAX_IMAGE_TITLE_LENGTH: 512;
readonly MAX_IMAGES_PER_URL: 1000;
readonly MAX_VIDEOS_PER_URL: 100;
readonly MAX_LINKS_PER_URL: 100;
readonly MAX_URL_ENTRIES: 50000;
readonly ISO_DATE_REGEX: RegExp;
readonly MAX_CUSTOM_NAMESPACES: 20;
readonly MAX_NAMESPACE_LENGTH: 512;
readonly MAX_PARSER_ERRORS: 100;
};
/**
* Default maximum number of items in each sitemap XML file
* Set below the max to leave room for URLs added during processing
*
* @see https://www.sitemaps.org/protocol.html#index
*/
export declare const DEFAULT_SITEMAP_ITEM_LIMIT = 45000;
node_modules/sitemap/dist/cjs/lib/constants.js
Generated Vendored
+63
View File
@@ -0,0 +1,63 @@
"use strict";
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
Object.defineProperty(exports, "__esModule", { value: true });
exports.DEFAULT_SITEMAP_ITEM_LIMIT = exports.LIMITS = void 0;
/**
* Shared constants used across the sitemap library
* This file serves as a single source of truth for limits and validation patterns
*/
/**
* Security limits for sitemap generation and parsing
*
* These limits are based on:
* - sitemaps.org protocol specification
* - Security best practices to prevent DoS and injection attacks
* - Google's sitemap extension specifications
*
* @see https://www.sitemaps.org/protocol.html
* @see https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap
*/
exports.LIMITS = {
// URL constraints per sitemaps.org spec
MAX_URL_LENGTH: 2048,
URL_PROTOCOL_REGEX: /^https?:\/\//i,
// Sitemap size limits per sitemaps.org spec
MIN_SITEMAP_ITEM_LIMIT: 1,
MAX_SITEMAP_ITEM_LIMIT: 50000,
// Video field length constraints per Google spec
MAX_VIDEO_TITLE_LENGTH: 100,
MAX_VIDEO_DESCRIPTION_LENGTH: 2048,
MAX_VIDEO_CATEGORY_LENGTH: 256,
MAX_TAGS_PER_VIDEO: 32,
// News field length constraints per Google spec
MAX_NEWS_TITLE_LENGTH: 200,
MAX_NEWS_NAME_LENGTH: 256,
// Image field length constraints per Google spec
MAX_IMAGE_CAPTION_LENGTH: 512,
MAX_IMAGE_TITLE_LENGTH: 512,
// Limits on number of items per URL entry
MAX_IMAGES_PER_URL: 1000,
MAX_VIDEOS_PER_URL: 100,
MAX_LINKS_PER_URL: 100,
// Total entries in a sitemap
MAX_URL_ENTRIES: 50000,
// Date validation - ISO 8601 / W3C format
ISO_DATE_REGEX: /^\d{4}-\d{2}-\d{2}(T\d{2}:\d{2}:\d{2}(\.\d{3})?([+-]\d{2}:\d{2}|Z)?)?$/,
// Custom namespace limits to prevent DoS
MAX_CUSTOM_NAMESPACES: 20,
MAX_NAMESPACE_LENGTH: 512,
// Cap on stored parser errors to prevent memory DoS (BB-03)
// Errors beyond this limit are counted in errorCount but not retained as objects
MAX_PARSER_ERRORS: 100,
};
/**
* Default maximum number of items in each sitemap XML file
* Set below the max to leave room for URLs added during processing
*
* @see https://www.sitemaps.org/protocol.html#index
*/
exports.DEFAULT_SITEMAP_ITEM_LIMIT = 45000;
node_modules/sitemap/dist/cjs/lib/errors.d.ts
Generated Vendored
+116
View File
@@ -0,0 +1,116 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
/**
* URL in SitemapItem does not exist
*/
export declare class NoURLError extends Error {
constructor(message?: string);
}
/**
* Config was not passed to SitemapItem constructor
*/
export declare class NoConfigError extends Error {
constructor(message?: string);
}
/**
* changefreq property in sitemap is invalid
*/
export declare class ChangeFreqInvalidError extends Error {
constructor(url: string, changefreq: any);
}
/**
* priority property in sitemap is invalid
*/
export declare class PriorityInvalidError extends Error {
constructor(url: string, priority: any);
}
/**
* SitemapIndex target Folder does not exists
*/
export declare class UndefinedTargetFolder extends Error {
constructor(message?: string);
}
export declare class InvalidVideoFormat extends Error {
constructor(url: string);
}
export declare class InvalidVideoDuration extends Error {
constructor(url: string, duration: any);
}
export declare class InvalidVideoDescription extends Error {
constructor(url: string, length: number);
}
export declare class InvalidVideoRating extends Error {
constructor(url: string, title: any, rating: any);
}
export declare class InvalidAttrValue extends Error {
constructor(key: string, val: any, validator: RegExp);
}
export declare class InvalidAttr extends Error {
constructor(key: string);
}
export declare class InvalidNewsFormat extends Error {
constructor(url: string);
}
export declare class InvalidNewsAccessValue extends Error {
constructor(url: string, access: any);
}
export declare class XMLLintUnavailable extends Error {
constructor(message?: string);
}
export declare class InvalidVideoTitle extends Error {
constructor(url: string, length: number);
}
export declare class InvalidVideoViewCount extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoTagCount extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoCategory extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoFamilyFriendly extends Error {
constructor(url: string, fam: string);
}
export declare class InvalidVideoRestriction extends Error {
constructor(url: string, code: string);
}
export declare class InvalidVideoRestrictionRelationship extends Error {
constructor(url: string, val?: string);
}
export declare class InvalidVideoPriceType extends Error {
constructor(url: string, priceType?: string, price?: string);
}
export declare class InvalidVideoResolution extends Error {
constructor(url: string, resolution: string);
}
export declare class InvalidVideoPriceCurrency extends Error {
constructor(url: string, currency: string);
}
export declare class EmptyStream extends Error {
constructor();
}
export declare class EmptySitemap extends Error {
constructor();
}
export declare class InvalidPathError extends Error {
constructor(path: string, reason: string);
}
export declare class InvalidHostnameError extends Error {
constructor(hostname: string, reason: string);
}
export declare class InvalidLimitError extends Error {
constructor(limit: any);
}
export declare class InvalidPublicBasePathError extends Error {
constructor(publicBasePath: string, reason: string);
}
export declare class InvalidXSLUrlError extends Error {
constructor(xslUrl: string, reason: string);
}
export declare class InvalidXMLAttributeNameError extends Error {
constructor(attributeName: string);
}
node_modules/sitemap/dist/cjs/lib/errors.js
Generated Vendored
+291
View File
@@ -0,0 +1,291 @@
"use strict";
/* eslint-disable @typescript-eslint/no-explicit-any */
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
Object.defineProperty(exports, "__esModule", { value: true });
exports.InvalidXMLAttributeNameError = exports.InvalidXSLUrlError = exports.InvalidPublicBasePathError = exports.InvalidLimitError = exports.InvalidHostnameError = exports.InvalidPathError = exports.EmptySitemap = exports.EmptyStream = exports.InvalidVideoPriceCurrency = exports.InvalidVideoResolution = exports.InvalidVideoPriceType = exports.InvalidVideoRestrictionRelationship = exports.InvalidVideoRestriction = exports.InvalidVideoFamilyFriendly = exports.InvalidVideoCategory = exports.InvalidVideoTagCount = exports.InvalidVideoViewCount = exports.InvalidVideoTitle = exports.XMLLintUnavailable = exports.InvalidNewsAccessValue = exports.InvalidNewsFormat = exports.InvalidAttr = exports.InvalidAttrValue = exports.InvalidVideoRating = exports.InvalidVideoDescription = exports.InvalidVideoDuration = exports.InvalidVideoFormat = exports.UndefinedTargetFolder = exports.PriorityInvalidError = exports.ChangeFreqInvalidError = exports.NoConfigError = exports.NoURLError = void 0;
/**
* URL in SitemapItem does not exist
*/
class NoURLError extends Error {
constructor(message) {
super(message || 'URL is required');
this.name = 'NoURLError';
Error.captureStackTrace(this, NoURLError);
}
}
exports.NoURLError = NoURLError;
/**
* Config was not passed to SitemapItem constructor
*/
class NoConfigError extends Error {
constructor(message) {
super(message || 'SitemapItem requires a configuration');
this.name = 'NoConfigError';
Error.captureStackTrace(this, NoConfigError);
}
}
exports.NoConfigError = NoConfigError;
/**
* changefreq property in sitemap is invalid
*/
class ChangeFreqInvalidError extends Error {
constructor(url, changefreq) {
super(`${url}: changefreq "${changefreq}" is invalid`);
this.name = 'ChangeFreqInvalidError';
Error.captureStackTrace(this, ChangeFreqInvalidError);
}
}
exports.ChangeFreqInvalidError = ChangeFreqInvalidError;
/**
* priority property in sitemap is invalid
*/
class PriorityInvalidError extends Error {
constructor(url, priority) {
super(`${url}: priority "${priority}" must be a number between 0 and 1 inclusive`);
this.name = 'PriorityInvalidError';
Error.captureStackTrace(this, PriorityInvalidError);
}
}
exports.PriorityInvalidError = PriorityInvalidError;
/**
* SitemapIndex target Folder does not exists
*/
class UndefinedTargetFolder extends Error {
constructor(message) {
super(message || 'Target folder must exist');
this.name = 'UndefinedTargetFolder';
Error.captureStackTrace(this, UndefinedTargetFolder);
}
}
exports.UndefinedTargetFolder = UndefinedTargetFolder;
class InvalidVideoFormat extends Error {
constructor(url) {
super(`${url} video must include thumbnail_loc, title and description fields for videos`);
this.name = 'InvalidVideoFormat';
Error.captureStackTrace(this, InvalidVideoFormat);
}
}
exports.InvalidVideoFormat = InvalidVideoFormat;
class InvalidVideoDuration extends Error {
constructor(url, duration) {
super(`${url} duration "${duration}" must be an integer of seconds between 0 and 28800`);
this.name = 'InvalidVideoDuration';
Error.captureStackTrace(this, InvalidVideoDuration);
}
}
exports.InvalidVideoDuration = InvalidVideoDuration;
class InvalidVideoDescription extends Error {
constructor(url, length) {
const message = `${url}: video description is too long ${length} vs limit of 2048 characters.`;
super(message);
this.name = 'InvalidVideoDescription';
Error.captureStackTrace(this, InvalidVideoDescription);
}
}
exports.InvalidVideoDescription = InvalidVideoDescription;
class InvalidVideoRating extends Error {
constructor(url, title, rating) {
super(`${url}: video "${title}" rating "${rating}" must be between 0 and 5 inclusive`);
this.name = 'InvalidVideoRating';
Error.captureStackTrace(this, InvalidVideoRating);
}
}
exports.InvalidVideoRating = InvalidVideoRating;
class InvalidAttrValue extends Error {
constructor(key, val, validator) {
super('"' +
val +
'" tested against: ' +
validator +
' is not a valid value for attr: "' +
key +
'"');
this.name = 'InvalidAttrValue';
Error.captureStackTrace(this, InvalidAttrValue);
}
}
exports.InvalidAttrValue = InvalidAttrValue;
// InvalidAttr is only thrown when attrbuilder is called incorrectly internally
/* istanbul ignore next */
class InvalidAttr extends Error {
constructor(key) {
super('"' + key + '" is malformed');
this.name = 'InvalidAttr';
Error.captureStackTrace(this, InvalidAttr);
}
}
exports.InvalidAttr = InvalidAttr;
class InvalidNewsFormat extends Error {
constructor(url) {
super(`${url} News must include publication, publication name, publication language, title, and publication_date for news`);
this.name = 'InvalidNewsFormat';
Error.captureStackTrace(this, InvalidNewsFormat);
}
}
exports.InvalidNewsFormat = InvalidNewsFormat;
class InvalidNewsAccessValue extends Error {
constructor(url, access) {
super(`${url} News access "${access}" must be either Registration, Subscription or not be present`);
this.name = 'InvalidNewsAccessValue';
Error.captureStackTrace(this, InvalidNewsAccessValue);
}
}
exports.InvalidNewsAccessValue = InvalidNewsAccessValue;
class XMLLintUnavailable extends Error {
constructor(message) {
super(message || 'xmlLint is not installed. XMLLint is required to validate');
this.name = 'XMLLintUnavailable';
Error.captureStackTrace(this, XMLLintUnavailable);
}
}
exports.XMLLintUnavailable = XMLLintUnavailable;
class InvalidVideoTitle extends Error {
constructor(url, length) {
super(`${url}: video title is too long ${length} vs 100 character limit`);
this.name = 'InvalidVideoTitle';
Error.captureStackTrace(this, InvalidVideoTitle);
}
}
exports.InvalidVideoTitle = InvalidVideoTitle;
class InvalidVideoViewCount extends Error {
constructor(url, count) {
super(`${url}: video view count must be positive, view count was ${count}`);
this.name = 'InvalidVideoViewCount';
Error.captureStackTrace(this, InvalidVideoViewCount);
}
}
exports.InvalidVideoViewCount = InvalidVideoViewCount;
class InvalidVideoTagCount extends Error {
constructor(url, count) {
super(`${url}: video can have no more than 32 tags, this has ${count}`);
this.name = 'InvalidVideoTagCount';
Error.captureStackTrace(this, InvalidVideoTagCount);
}
}
exports.InvalidVideoTagCount = InvalidVideoTagCount;
class InvalidVideoCategory extends Error {
constructor(url, count) {
super(`${url}: video category can only be 256 characters but was passed ${count}`);
this.name = 'InvalidVideoCategory';
Error.captureStackTrace(this, InvalidVideoCategory);
}
}
exports.InvalidVideoCategory = InvalidVideoCategory;
class InvalidVideoFamilyFriendly extends Error {
constructor(url, fam) {
super(`${url}: video family friendly must be yes or no, was passed "${fam}"`);
this.name = 'InvalidVideoFamilyFriendly';
Error.captureStackTrace(this, InvalidVideoFamilyFriendly);
}
}
exports.InvalidVideoFamilyFriendly = InvalidVideoFamilyFriendly;
class InvalidVideoRestriction extends Error {
constructor(url, code) {
super(`${url}: video restriction must be one or more two letter country codes. Was passed "${code}"`);
this.name = 'InvalidVideoRestriction';
Error.captureStackTrace(this, InvalidVideoRestriction);
}
}
exports.InvalidVideoRestriction = InvalidVideoRestriction;
class InvalidVideoRestrictionRelationship extends Error {
constructor(url, val) {
super(`${url}: video restriction relationship must be either allow or deny. Was passed "${val}"`);
this.name = 'InvalidVideoRestrictionRelationship';
Error.captureStackTrace(this, InvalidVideoRestrictionRelationship);
}
}
exports.InvalidVideoRestrictionRelationship = InvalidVideoRestrictionRelationship;
class InvalidVideoPriceType extends Error {
constructor(url, priceType, price) {
super(priceType === undefined && price === ''
? `${url}: video priceType is required when price is not provided`
: `${url}: video price type "${priceType}" is not "rent" or "purchase"`);
this.name = 'InvalidVideoPriceType';
Error.captureStackTrace(this, InvalidVideoPriceType);
}
}
exports.InvalidVideoPriceType = InvalidVideoPriceType;
class InvalidVideoResolution extends Error {
constructor(url, resolution) {
super(`${url}: video price resolution "${resolution}" is not hd or sd`);
this.name = 'InvalidVideoResolution';
Error.captureStackTrace(this, InvalidVideoResolution);
}
}
exports.InvalidVideoResolution = InvalidVideoResolution;
class InvalidVideoPriceCurrency extends Error {
constructor(url, currency) {
super(`${url}: video price currency "${currency}" must be a three capital letter abbrieviation for the country currency`);
this.name = 'InvalidVideoPriceCurrency';
Error.captureStackTrace(this, InvalidVideoPriceCurrency);
}
}
exports.InvalidVideoPriceCurrency = InvalidVideoPriceCurrency;
class EmptyStream extends Error {
constructor() {
super('You have ended the stream before anything was written. streamToPromise MUST be called before ending the stream.');
this.name = 'EmptyStream';
Error.captureStackTrace(this, EmptyStream);
}
}
exports.EmptyStream = EmptyStream;
class EmptySitemap extends Error {
constructor() {
super('You ended the stream without writing anything.');
this.name = 'EmptySitemap';
Error.captureStackTrace(this, EmptyStream);
}
}
exports.EmptySitemap = EmptySitemap;
class InvalidPathError extends Error {
constructor(path, reason) {
super(`Invalid path "${path}": ${reason}`);
this.name = 'InvalidPathError';
Error.captureStackTrace(this, InvalidPathError);
}
}
exports.InvalidPathError = InvalidPathError;
class InvalidHostnameError extends Error {
constructor(hostname, reason) {
super(`Invalid hostname "${hostname}": ${reason}`);
this.name = 'InvalidHostnameError';
Error.captureStackTrace(this, InvalidHostnameError);
}
}
exports.InvalidHostnameError = InvalidHostnameError;
class InvalidLimitError extends Error {
constructor(limit) {
super(`Invalid limit "${limit}": must be a number between 1 and 50000 (per sitemaps.org spec)`);
this.name = 'InvalidLimitError';
Error.captureStackTrace(this, InvalidLimitError);
}
}
exports.InvalidLimitError = InvalidLimitError;
class InvalidPublicBasePathError extends Error {
constructor(publicBasePath, reason) {
super(`Invalid publicBasePath "${publicBasePath}": ${reason}`);
this.name = 'InvalidPublicBasePathError';
Error.captureStackTrace(this, InvalidPublicBasePathError);
}
}
exports.InvalidPublicBasePathError = InvalidPublicBasePathError;
class InvalidXSLUrlError extends Error {
constructor(xslUrl, reason) {
super(`Invalid xslUrl "${xslUrl}": ${reason}`);
this.name = 'InvalidXSLUrlError';
Error.captureStackTrace(this, InvalidXSLUrlError);
}
}
exports.InvalidXSLUrlError = InvalidXSLUrlError;
class InvalidXMLAttributeNameError extends Error {
constructor(attributeName) {
super(`Invalid XML attribute name "${attributeName}": must contain only alphanumeric characters, hyphens, underscores, and colons`);
this.name = 'InvalidXMLAttributeNameError';
Error.captureStackTrace(this, InvalidXMLAttributeNameError);
}
}
exports.InvalidXMLAttributeNameError = InvalidXMLAttributeNameError;
node_modules/sitemap/dist/cjs/lib/sitemap-index-parser.d.ts
Generated Vendored
+55
View File
@@ -0,0 +1,55 @@
import type { SAXStream } from 'sax';
import { Readable, Transform, TransformOptions, TransformCallback } from 'node:stream';
import { IndexItem, ErrorLevel } from './types.js';
type Logger = (level: 'warn' | 'error' | 'info' | 'log', ...message: Parameters<Console['log']>) => void;
export interface XMLToSitemapIndexItemStreamOptions extends TransformOptions {
level?: ErrorLevel;
logger?: Logger | false;
}
/**
* Takes a stream of xml and transforms it into a stream of IndexItems
* Use this to parse existing sitemap indices into config options compatible with this library
*/
export declare class XMLToSitemapIndexStream extends Transform {
level: ErrorLevel;
logger: Logger;
error: Error | null;
saxStream: SAXStream;
constructor(opts?: XMLToSitemapIndexItemStreamOptions);
_transform(data: string, encoding: string, callback: TransformCallback): void;
private err;
}
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
const { createReadStream } = require('fs')
const { parseSitemapIndex, createSitemap } = require('sitemap')
parseSitemapIndex(createReadStream('./example-index.xml')).then(
// produces the same xml
// you can, of course, more practically modify it or store it
(xmlConfig) => console.log(createSitemap(xmlConfig).toString()),
(err) => console.log(err)
)
```
@param {Readable} xml what to parse
@param {number} maxEntries Maximum number of sitemap entries to parse (default: 50,000 per sitemaps.org spec)
@return {Promise<IndexItem[]>} resolves with list of index items that can be fed into a SitemapIndexStream. Rejects with an Error object.
*/
export declare function parseSitemapIndex(xml: Readable, maxEntries?: number): Promise<IndexItem[]>;
export interface IndexObjectStreamToJSONOptions extends TransformOptions {
lineSeparated: boolean;
}
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
export declare class IndexObjectStreamToJSON extends Transform {
lineSeparated: boolean;
firstWritten: boolean;
constructor(opts?: IndexObjectStreamToJSONOptions);
_transform(chunk: IndexItem, encoding: string, cb: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
export {};
node_modules/sitemap/dist/cjs/lib/sitemap-index-parser.js
Generated Vendored
+271
View File
@@ -0,0 +1,271 @@
"use strict";
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
exports.IndexObjectStreamToJSON = exports.XMLToSitemapIndexStream = void 0;
exports.parseSitemapIndex = parseSitemapIndex;
const sax_1 = __importDefault(require("sax"));
const node_stream_1 = require("node:stream");
const types_js_1 = require("./types.js");
const validation_js_1 = require("./validation.js");
const constants_js_1 = require("./constants.js");
function isValidTagName(tagName) {
// This only works because the enum name and value are the same
return tagName in types_js_1.IndexTagNames;
}
function tagTemplate() {
return {
url: '',
};
}
const defaultLogger = (level, ...message) => console[level](...message);
const defaultStreamOpts = {
logger: defaultLogger,
};
/**
* Takes a stream of xml and transforms it into a stream of IndexItems
* Use this to parse existing sitemap indices into config options compatible with this library
*/
class XMLToSitemapIndexStream extends node_stream_1.Transform {
level;
logger;
error;
saxStream;
constructor(opts = defaultStreamOpts) {
opts.objectMode = true;
super(opts);
this.error = null;
this.saxStream = sax_1.default.createStream(true, {
xmlns: true,
// @ts-expect-error - SAX types don't include strictEntities option
strictEntities: true,
trim: true,
});
this.level = opts.level || types_js_1.ErrorLevel.WARN;
if (this.level !== types_js_1.ErrorLevel.SILENT && opts.logger !== false) {
this.logger = opts.logger ?? defaultLogger;
}
else {
this.logger = () => undefined;
}
let currentItem = tagTemplate();
let currentTag;
this.saxStream.on('opentagstart', (tag) => {
currentTag = tag.name;
});
this.saxStream.on('opentag', (tag) => {
if (!isValidTagName(tag.name)) {
this.logger('warn', 'unhandled tag', tag.name);
this.err(`unhandled tag: ${tag.name}`);
}
});
this.saxStream.on('text', (text) => {
switch (currentTag) {
case types_js_1.IndexTagNames.loc:
// Validate URL for security: prevents protocol injection, checks length limits
try {
(0, validation_js_1.validateURL)(text, 'Sitemap index URL');
currentItem.url = text;
}
catch (error) {
const errMsg = error instanceof Error ? error.message : String(error);
this.logger('warn', 'Invalid URL in sitemap index:', errMsg);
this.err(`Invalid URL in sitemap index: ${errMsg}`);
}
break;
case types_js_1.IndexTagNames.lastmod:
// Validate date format for security and spec compliance
if (text && !constants_js_1.LIMITS.ISO_DATE_REGEX.test(text)) {
this.logger('warn', 'Invalid lastmod date format in sitemap index:', text);
this.err(`Invalid lastmod date format: ${text}`);
}
else {
currentItem.lastmod = text;
}
break;
default:
this.logger('log', 'unhandled text for tag:', currentTag, `'${text}'`);
this.err(`unhandled text for tag: ${currentTag} '${text}'`);
break;
}
});
this.saxStream.on('cdata', (text) => {
switch (currentTag) {
case types_js_1.IndexTagNames.loc:
// Validate URL for security: prevents protocol injection, checks length limits
try {
(0, validation_js_1.validateURL)(text, 'Sitemap index URL');
currentItem.url = text;
}
catch (error) {
const errMsg = error instanceof Error ? error.message : String(error);
this.logger('warn', 'Invalid URL in sitemap index:', errMsg);
this.err(`Invalid URL in sitemap index: ${errMsg}`);
}
break;
case types_js_1.IndexTagNames.lastmod:
// Validate date format for security and spec compliance
if (text && !constants_js_1.LIMITS.ISO_DATE_REGEX.test(text)) {
this.logger('warn', 'Invalid lastmod date format in sitemap index:', text);
this.err(`Invalid lastmod date format: ${text}`);
}
else {
currentItem.lastmod = text;
}
break;
default:
this.logger('log', 'unhandled cdata for tag:', currentTag);
this.err(`unhandled cdata for tag: ${currentTag}`);
break;
}
});
this.saxStream.on('attribute', (attr) => {
switch (currentTag) {
case types_js_1.IndexTagNames.sitemapindex:
break;
default:
this.logger('log', 'unhandled attr', currentTag, attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
});
this.saxStream.on('closetag', (tag) => {
switch (tag) {
case types_js_1.IndexTagNames.sitemap:
// Only push items with valid URLs (non-empty after validation)
if (currentItem.url) {
this.push(currentItem);
}
currentItem = tagTemplate();
break;
default:
break;
}
});
}
_transform(data, encoding, callback) {
try {
const cb = () => callback(this.level === types_js_1.ErrorLevel.THROW ? this.error : null);
// correcting the type here can be done without making it a breaking change
// TODO fix this
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
if (!this.saxStream.write(data, encoding)) {
this.saxStream.once('drain', cb);
}
else {
process.nextTick(cb);
}
}
catch (error) {
callback(error);
}
}
err(msg) {
if (!this.error)
this.error = new Error(msg);
}
}
exports.XMLToSitemapIndexStream = XMLToSitemapIndexStream;
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
const { createReadStream } = require('fs')
const { parseSitemapIndex, createSitemap } = require('sitemap')
parseSitemapIndex(createReadStream('./example-index.xml')).then(
// produces the same xml
// you can, of course, more practically modify it or store it
(xmlConfig) => console.log(createSitemap(xmlConfig).toString()),
(err) => console.log(err)
)
```
@param {Readable} xml what to parse
@param {number} maxEntries Maximum number of sitemap entries to parse (default: 50,000 per sitemaps.org spec)
@return {Promise<IndexItem[]>} resolves with list of index items that can be fed into a SitemapIndexStream. Rejects with an Error object.
*/
async function parseSitemapIndex(xml, maxEntries = constants_js_1.LIMITS.MAX_SITEMAP_ITEM_LIMIT) {
const urls = [];
return new Promise((resolve, reject) => {
let settled = false;
const parser = new XMLToSitemapIndexStream();
// Handle source stream errors (prevents unhandled error events on xml)
xml.on('error', (error) => {
if (!settled) {
settled = true;
reject(error);
}
});
xml
.pipe(parser)
.on('data', (smi) => {
if (settled)
return;
// Security: Prevent memory exhaustion by limiting number of entries
if (urls.length >= maxEntries) {
settled = true;
reject(new Error(`Sitemap index exceeds maximum allowed entries (${maxEntries})`));
// Immediately destroy both streams to stop further processing (BB-05)
parser.destroy();
xml.destroy();
return;
}
urls.push(smi);
})
.on('end', () => {
if (!settled) {
settled = true;
resolve(urls);
}
})
.on('error', (error) => {
if (!settled) {
settled = true;
reject(error);
}
});
});
}
const defaultObjectStreamOpts = {
lineSeparated: false,
};
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
class IndexObjectStreamToJSON extends node_stream_1.Transform {
lineSeparated;
firstWritten;
constructor(opts = defaultObjectStreamOpts) {
opts.writableObjectMode = true;
super(opts);
this.lineSeparated = opts.lineSeparated;
this.firstWritten = false;
}
_transform(chunk, encoding, cb) {
if (!this.firstWritten) {
this.firstWritten = true;
if (!this.lineSeparated) {
this.push('[');
}
}
else if (this.lineSeparated) {
this.push('\n');
}
else {
this.push(',');
}
if (chunk) {
this.push(JSON.stringify(chunk));
}
cb();
}
_flush(cb) {
if (!this.lineSeparated) {
this.push(']');
}
cb();
}
}
exports.IndexObjectStreamToJSON = IndexObjectStreamToJSON;
node_modules/sitemap/dist/cjs/lib/sitemap-index-stream.d.ts
Generated Vendored
+169
View File
@@ -0,0 +1,169 @@
import { WriteStream } from 'node:fs';
import { Transform, TransformOptions, TransformCallback } from 'node:stream';
import { IndexItem, SitemapItemLoose, ErrorLevel, IndexTagNames } from './types.js';
import { SitemapStream } from './sitemap-stream.js';
export { IndexTagNames };
/**
* Options for the SitemapIndexStream
*/
export interface SitemapIndexStreamOptions extends TransformOptions {
/**
* Whether to output the lastmod date only (no time)
*
* @default false
*/
lastmodDateOnly?: boolean;
/**
* How to handle errors in passed in urls
*
* @default ErrorLevel.WARN
*/
level?: ErrorLevel;
/**
* URL to an XSL stylesheet to include in the XML
*/
xslUrl?: string;
}
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {Transform}
*/
export declare class SitemapIndexStream extends Transform {
lastmodDateOnly: boolean;
level: ErrorLevel;
xslUrl?: string;
private hasHeadOutput;
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapIndexStreamOptions} [opts=defaultStreamOpts] - Stream options.
*/
constructor(opts?: SitemapIndexStreamOptions);
private writeHeadOutput;
_transform(item: IndexItem | string, encoding: string, callback: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
/**
* Callback function type for creating new sitemap streams when the item limit is reached.
*
* This function is called by SitemapAndIndexStream to create a new sitemap file when
* the current one reaches the item limit.
*
* @param i - The zero-based index of the sitemap file being created (0 for first sitemap,
* 1 for second, etc.)
* @returns A tuple containing:
* - [0]: IndexItem or URL string to add to the sitemap index
* - [1]: SitemapStream instance for writing sitemap items
* - [2]: WriteStream where the sitemap will be piped (the stream will be
* awaited for 'finish' before creating the next sitemap)
*
* @example
* ```typescript
* const getSitemapStream = (i: number) => {
* const sitemapStream = new SitemapStream();
* const path = `./sitemap-${i}.xml`;
* const writeStream = createWriteStream(path);
* sitemapStream.pipe(writeStream);
* return [`https://example.com/${path}`, sitemapStream, writeStream];
* };
* ```
*/
type getSitemapStreamFunc = (i: number) => [IndexItem | string, SitemapStream, WriteStream];
/**
* Options for the SitemapAndIndexStream
*
* @extends {SitemapIndexStreamOptions}
*/
export interface SitemapAndIndexStreamOptions extends SitemapIndexStreamOptions {
/**
* Max number of items in each sitemap XML file.
*
* When the limit is reached the current sitemap file will be closed,
* a wait for `finish` on the target write stream will happen,
* and a new sitemap file will be created.
*
* Range: 1 - 50,000
*
* @default 45000
*/
limit?: number;
/**
* Callback for SitemapIndexAndStream that creates a new sitemap stream for a given sitemap index.
*
* Called when a new sitemap file is needed.
*
* The write stream is the destination where the sitemap was piped.
* SitemapAndIndexStream will wait for the `finish` event on each sitemap's
* write stream before moving on to the next sitemap. This ensures that the
* contents of the write stream will be fully written before being used
* by any following operations (e.g. uploading, reading contents for unit tests).
*
* @param i - The index of the sitemap file
* @returns A tuple containing the index item to be written into the sitemap index, the sitemap stream, and the write stream for the sitemap pipe destination
*/
getSitemapStream: getSitemapStreamFunc;
}
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {SitemapIndexStream}
*/
export declare class SitemapAndIndexStream extends SitemapIndexStream {
private itemsWritten;
private getSitemapStream;
private currentSitemap?;
private limit;
private currentSitemapPipeline?;
/**
* Flag to prevent race conditions when creating new sitemap files.
* Set to true while waiting for the current sitemap to finish and
* a new one to be created.
*/
private isCreatingSitemap;
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapAndIndexStreamOptions} opts - Stream options.
*/
constructor(opts: SitemapAndIndexStreamOptions);
_transform(item: SitemapItemLoose, encoding: string, callback: TransformCallback): void;
private writeItem;
/**
* Called when the stream is finished.
* If there is a current sitemap, we wait for it to finish before calling the callback.
* Includes proper event listener cleanup to prevent memory leaks.
*
* @param cb - The callback to invoke when flushing is complete
*/
_flush(cb: TransformCallback): void;
private createSitemap;
}
node_modules/sitemap/dist/cjs/lib/sitemap-index-stream.js
Generated Vendored
+363
View File
@@ -0,0 +1,363 @@
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.SitemapAndIndexStream = exports.SitemapIndexStream = exports.IndexTagNames = void 0;
const node_stream_1 = require("node:stream");
const types_js_1 = require("./types.js");
Object.defineProperty(exports, "IndexTagNames", { enumerable: true, get: function () { return types_js_1.IndexTagNames; } });
const sitemap_stream_js_1 = require("./sitemap-stream.js");
const sitemap_xml_js_1 = require("./sitemap-xml.js");
const constants_js_1 = require("./constants.js");
const validation_js_1 = require("./validation.js");
const xmlDec = '<?xml version="1.0" encoding="UTF-8"?>';
const sitemapIndexTagStart = '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">';
const closetag = '</sitemapindex>';
const defaultStreamOpts = {};
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {Transform}
*/
class SitemapIndexStream extends node_stream_1.Transform {
lastmodDateOnly;
level;
xslUrl;
hasHeadOutput;
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapIndexStreamOptions} [opts=defaultStreamOpts] - Stream options.
*/
constructor(opts = defaultStreamOpts) {
opts.objectMode = true;
super(opts);
this.hasHeadOutput = false;
this.lastmodDateOnly = opts.lastmodDateOnly || false;
this.level = opts.level ?? types_js_1.ErrorLevel.WARN;
if (opts.xslUrl !== undefined) {
(0, validation_js_1.validateXSLUrl)(opts.xslUrl);
}
this.xslUrl = opts.xslUrl;
}
writeHeadOutput() {
this.hasHeadOutput = true;
let stylesheet = '';
if (this.xslUrl) {
stylesheet = (0, sitemap_stream_js_1.stylesheetInclude)(this.xslUrl);
}
this.push(xmlDec + stylesheet + sitemapIndexTagStart);
}
_transform(item, encoding, callback) {
if (!this.hasHeadOutput) {
this.writeHeadOutput();
}
try {
// Validate URL using centralized validation (checks protocol, length, format)
const url = typeof item === 'string' ? item : item.url;
if (!url || typeof url !== 'string') {
const error = new Error('Invalid sitemap index item: URL must be a non-empty string');
if (this.level === types_js_1.ErrorLevel.THROW) {
callback(error);
return;
}
else if (this.level === types_js_1.ErrorLevel.WARN) {
console.warn(error.message, item);
}
// For SILENT or after WARN, skip this item
callback();
return;
}
// Security: Use centralized validation to enforce protocol restrictions,
// length limits, and prevent injection attacks
try {
(0, validation_js_1.validateURL)(url, 'Sitemap index URL');
}
catch (error) {
// Wrap the validation error with consistent message format
const validationMsg = error instanceof Error ? error.message : String(error);
const err = new Error(`Invalid URL in sitemap index: ${validationMsg}`);
if (this.level === types_js_1.ErrorLevel.THROW) {
callback(err);
return;
}
else if (this.level === types_js_1.ErrorLevel.WARN) {
console.warn(err.message);
}
// For SILENT or after WARN, skip this item
callback();
return;
}
this.push((0, sitemap_xml_js_1.otag)(types_js_1.IndexTagNames.sitemap));
if (typeof item === 'string') {
this.push((0, sitemap_xml_js_1.element)(types_js_1.IndexTagNames.loc, item));
}
else {
this.push((0, sitemap_xml_js_1.element)(types_js_1.IndexTagNames.loc, item.url));
if (item.lastmod) {
try {
const lastmod = new Date(item.lastmod).toISOString();
this.push((0, sitemap_xml_js_1.element)(types_js_1.IndexTagNames.lastmod, this.lastmodDateOnly ? lastmod.slice(0, 10) : lastmod));
}
catch {
const error = new Error(`Invalid lastmod date in sitemap index: ${item.lastmod}`);
if (this.level === types_js_1.ErrorLevel.THROW) {
callback(error);
return;
}
else if (this.level === types_js_1.ErrorLevel.WARN) {
console.warn(error.message);
}
// Continue without lastmod for SILENT or after WARN
}
}
}
this.push((0, sitemap_xml_js_1.ctag)(types_js_1.IndexTagNames.sitemap));
callback();
}
catch (error) {
callback(error instanceof Error ? error : new Error(String(error)));
}
}
_flush(cb) {
if (!this.hasHeadOutput) {
this.writeHeadOutput();
}
this.push(closetag);
cb();
}
}
exports.SitemapIndexStream = SitemapIndexStream;
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {SitemapIndexStream}
*/
class SitemapAndIndexStream extends SitemapIndexStream {
itemsWritten;
getSitemapStream;
currentSitemap;
limit;
currentSitemapPipeline;
/**
* Flag to prevent race conditions when creating new sitemap files.
* Set to true while waiting for the current sitemap to finish and
* a new one to be created.
*/
isCreatingSitemap;
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapAndIndexStreamOptions} opts - Stream options.
*/
constructor(opts) {
opts.objectMode = true;
super(opts);
this.itemsWritten = 0;
this.getSitemapStream = opts.getSitemapStream;
this.limit = opts.limit ?? constants_js_1.DEFAULT_SITEMAP_ITEM_LIMIT;
this.isCreatingSitemap = false;
// Validate limit is within acceptable range per sitemaps.org spec
// See: https://www.sitemaps.org/protocol.html#index
if (this.limit < constants_js_1.LIMITS.MIN_SITEMAP_ITEM_LIMIT ||
this.limit > constants_js_1.LIMITS.MAX_SITEMAP_ITEM_LIMIT) {
throw new Error(`limit must be between ${constants_js_1.LIMITS.MIN_SITEMAP_ITEM_LIMIT} and ${constants_js_1.LIMITS.MAX_SITEMAP_ITEM_LIMIT} per sitemaps.org spec, got ${this.limit}`);
}
}
_transform(item, encoding, callback) {
if (this.itemsWritten % this.limit === 0) {
// Prevent race condition if multiple items arrive during sitemap creation
if (this.isCreatingSitemap) {
// Wait and retry on next tick
process.nextTick(() => this._transform(item, encoding, callback));
return;
}
if (this.currentSitemap) {
this.isCreatingSitemap = true;
const currentSitemap = this.currentSitemap;
const currentPipeline = this.currentSitemapPipeline;
// Set up promises with proper cleanup to prevent memory leaks
const onFinish = new Promise((resolve, reject) => {
const finishHandler = () => {
currentSitemap.off('error', errorHandler);
resolve();
};
const errorHandler = (err) => {
currentSitemap.off('finish', finishHandler);
reject(err);
};
currentSitemap.on('finish', finishHandler);
currentSitemap.on('error', errorHandler);
currentSitemap.end();
});
const onPipelineFinish = currentPipeline
? new Promise((resolve, reject) => {
const finishHandler = () => {
currentPipeline.off('error', errorHandler);
resolve();
};
const errorHandler = (err) => {
currentPipeline.off('finish', finishHandler);
reject(err);
};
currentPipeline.on('finish', finishHandler);
currentPipeline.on('error', errorHandler);
})
: Promise.resolve();
Promise.all([onFinish, onPipelineFinish])
.then(() => {
this.isCreatingSitemap = false;
this.createSitemap(encoding);
this.writeItem(item, callback);
})
.catch((err) => {
this.isCreatingSitemap = false;
callback(err);
});
return;
}
else {
this.createSitemap(encoding);
}
}
this.writeItem(item, callback);
}
writeItem(item, callback) {
if (!this.currentSitemap) {
callback(new Error('No sitemap stream available'));
return;
}
if (!this.currentSitemap.write(item)) {
this.currentSitemap.once('drain', callback);
}
else {
process.nextTick(callback);
}
// Increment the count of items written
this.itemsWritten++;
}
/**
* Called when the stream is finished.
* If there is a current sitemap, we wait for it to finish before calling the callback.
* Includes proper event listener cleanup to prevent memory leaks.
*
* @param cb - The callback to invoke when flushing is complete
*/
_flush(cb) {
const currentSitemap = this.currentSitemap;
const currentPipeline = this.currentSitemapPipeline;
const onFinish = new Promise((resolve, reject) => {
if (currentSitemap) {
const finishHandler = () => {
currentSitemap.off('error', errorHandler);
resolve();
};
const errorHandler = (err) => {
currentSitemap.off('finish', finishHandler);
reject(err);
};
currentSitemap.on('finish', finishHandler);
currentSitemap.on('error', errorHandler);
currentSitemap.end();
}
else {
resolve();
}
});
const onPipelineFinish = new Promise((resolve, reject) => {
if (currentPipeline) {
const finishHandler = () => {
currentPipeline.off('error', errorHandler);
resolve();
};
const errorHandler = (err) => {
currentPipeline.off('finish', finishHandler);
reject(err);
};
currentPipeline.on('finish', finishHandler);
currentPipeline.on('error', errorHandler);
// The pipeline (pipe target) will get its end() call
// from the sitemap stream ending.
}
else {
resolve();
}
});
Promise.all([onFinish, onPipelineFinish])
.then(() => {
super._flush(cb);
})
.catch((err) => {
cb(err);
});
}
createSitemap(encoding) {
const sitemapIndex = this.itemsWritten / this.limit;
let result;
try {
result = this.getSitemapStream(sitemapIndex);
}
catch (err) {
this.emit('error', new Error(`getSitemapStream callback threw an error for index ${sitemapIndex}: ${err instanceof Error ? err.message : String(err)}`));
return;
}
// Validate the return value
if (!Array.isArray(result) || result.length !== 3) {
this.emit('error', new Error(`getSitemapStream must return a 3-element array [IndexItem | string, SitemapStream, WriteStream], got: ${typeof result}`));
return;
}
const [idxItem, currentSitemap, currentSitemapPipeline] = result;
// Validate each element
if (!idxItem ||
(typeof idxItem !== 'string' && typeof idxItem !== 'object')) {
this.emit('error', new Error('getSitemapStream must return an IndexItem or string as the first element'));
return;
}
if (!currentSitemap || typeof currentSitemap.write !== 'function') {
this.emit('error', new Error('getSitemapStream must return a SitemapStream as the second element'));
return;
}
if (currentSitemapPipeline &&
typeof currentSitemapPipeline.write !== 'function') {
this.emit('error', new Error('getSitemapStream must return a WriteStream or undefined as the third element'));
return;
}
// Propagate errors from the sitemap stream
currentSitemap.on('error', (err) => this.emit('error', err));
this.currentSitemap = currentSitemap;
this.currentSitemapPipeline = currentSitemapPipeline;
super._transform(idxItem, encoding, () => {
// We are not too concerned about waiting for the index item to be written
// as we'll wait for the file to finish at the end, and index file write
// volume tends to be small in comparison to sitemap writes.
// noop
});
}
}
exports.SitemapAndIndexStream = SitemapAndIndexStream;
node_modules/sitemap/dist/cjs/lib/sitemap-item-stream.d.ts
Generated Vendored
+21
View File
@@ -0,0 +1,21 @@
import { Transform, TransformOptions, TransformCallback } from 'node:stream';
import { SitemapItem, ErrorLevel } from './types.js';
export interface SitemapItemStreamOptions extends TransformOptions {
level?: ErrorLevel;
}
/**
* Takes a stream of SitemapItemOptions and spits out xml for each
* @example
* // writes <url><loc>https://example.com</loc><url><url><loc>https://example.com/2</loc><url>
* const smis = new SitemapItemStream({level: 'warn'})
* smis.pipe(writestream)
* smis.write({url: 'https://example.com', img: [], video: [], links: []})
* smis.write({url: 'https://example.com/2', img: [], video: [], links: []})
* smis.end()
* @param level - Error level
*/
export declare class SitemapItemStream extends Transform {
level: ErrorLevel;
constructor(opts?: SitemapItemStreamOptions);
_transform(item: SitemapItem, encoding: string, callback: TransformCallback): void;
}
node_modules/sitemap/dist/cjs/lib/sitemap-item-stream.js
Generated Vendored
+208
View File
@@ -0,0 +1,208 @@
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.SitemapItemStream = void 0;
const node_stream_1 = require("node:stream");
const errors_js_1 = require("./errors.js");
const types_js_1 = require("./types.js");
const sitemap_xml_js_1 = require("./sitemap-xml.js");
/**
* Builds an attributes object for XML elements from configuration object
* Extracts attributes based on colon-delimited keys (e.g., 'price:currency' -> { currency: value })
*
* @param conf - Configuration object containing attribute values
* @param keys - Single key or array of keys in format 'namespace:attribute'
* @returns Record of attribute names to string values (may contain non-string values from conf)
* @throws {InvalidAttr} When key format is invalid (must contain exactly one colon)
*
* @example
* attrBuilder({ 'price:currency': 'USD', 'price:type': 'rent' }, ['price:currency', 'price:type'])
* // Returns: { currency: 'USD', type: 'rent' }
*/
function attrBuilder(conf, keys) {
if (typeof keys === 'string') {
keys = [keys];
}
const iv = {};
return keys.reduce((attrs, key) => {
if (conf[key] !== undefined) {
const keyAr = key.split(':');
if (keyAr.length !== 2) {
throw new errors_js_1.InvalidAttr(key);
}
attrs[keyAr[1]] = conf[key];
}
return attrs;
}, iv);
}
/**
* Takes a stream of SitemapItemOptions and spits out xml for each
* @example
* // writes <url><loc>https://example.com</loc><url><url><loc>https://example.com/2</loc><url>
* const smis = new SitemapItemStream({level: 'warn'})
* smis.pipe(writestream)
* smis.write({url: 'https://example.com', img: [], video: [], links: []})
* smis.write({url: 'https://example.com/2', img: [], video: [], links: []})
* smis.end()
* @param level - Error level
*/
class SitemapItemStream extends node_stream_1.Transform {
level;
constructor(opts = { level: types_js_1.ErrorLevel.WARN }) {
opts.objectMode = true;
super(opts);
this.level = opts.level || types_js_1.ErrorLevel.WARN;
}
_transform(item, encoding, callback) {
this.push((0, sitemap_xml_js_1.otag)(types_js_1.TagNames.url));
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames.loc, item.url));
if (item.lastmod) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames.lastmod, item.lastmod));
}
if (item.changefreq) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames.changefreq, item.changefreq));
}
if (item.priority !== undefined && item.priority !== null) {
if (item.fullPrecisionPriority) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames.priority, item.priority.toString()));
}
else {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames.priority, item.priority.toFixed(1)));
}
}
item.video.forEach((video) => {
this.push((0, sitemap_xml_js_1.otag)(types_js_1.TagNames['video:video']));
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:thumbnail_loc'], video.thumbnail_loc));
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:title'], video.title));
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:description'], video.description));
if (video.content_loc) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:content_loc'], video.content_loc));
}
if (video.player_loc) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:player_loc'], attrBuilder(video, [
'player_loc:autoplay',
'player_loc:allow_embed',
]), video.player_loc));
}
if (video.duration) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:duration'], video.duration.toString()));
}
if (video.expiration_date) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:expiration_date'], video.expiration_date));
}
if (video.rating !== undefined) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:rating'], video.rating.toString()));
}
if (video.view_count !== undefined) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:view_count'], String(video.view_count)));
}
if (video.publication_date) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:publication_date'], video.publication_date));
}
if (video.tag && video.tag.length > 0) {
for (const tag of video.tag) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:tag'], tag));
}
}
if (video.category) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:category'], video.category));
}
if (video.family_friendly) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:family_friendly'], video.family_friendly));
}
if (video.restriction) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:restriction'], attrBuilder(video, 'restriction:relationship'), video.restriction));
}
if (video.gallery_loc) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:gallery_loc'], attrBuilder(video, 'gallery_loc:title'), video.gallery_loc));
}
if (video.price) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:price'], attrBuilder(video, [
'price:resolution',
'price:currency',
'price:type',
]), video.price));
}
if (video.requires_subscription) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:requires_subscription'], video.requires_subscription));
}
if (video.uploader) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:uploader'], attrBuilder(video, 'uploader:info'), video.uploader));
}
if (video.platform) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:platform'], attrBuilder(video, 'platform:relationship'), video.platform));
}
if (video.live) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:live'], video.live));
}
if (video.id) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['video:id'], { type: 'url' }, video.id));
}
this.push((0, sitemap_xml_js_1.ctag)(types_js_1.TagNames['video:video']));
});
item.links.forEach((link) => {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['xhtml:link'], {
rel: 'alternate',
hreflang: link.lang || link.hreflang,
href: link.url,
}));
});
if (item.expires) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames.expires, new Date(item.expires).toISOString()));
}
if (item.androidLink) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['xhtml:link'], {
rel: 'alternate',
href: item.androidLink,
}));
}
if (item.ampLink) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['xhtml:link'], {
rel: 'amphtml',
href: item.ampLink,
}));
}
if (item.news) {
this.push((0, sitemap_xml_js_1.otag)(types_js_1.TagNames['news:news']));
this.push((0, sitemap_xml_js_1.otag)(types_js_1.TagNames['news:publication']));
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['news:name'], item.news.publication.name));
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['news:language'], item.news.publication.language));
this.push((0, sitemap_xml_js_1.ctag)(types_js_1.TagNames['news:publication']));
if (item.news.access) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['news:access'], item.news.access));
}
if (item.news.genres) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['news:genres'], item.news.genres));
}
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['news:publication_date'], item.news.publication_date));
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['news:title'], item.news.title));
if (item.news.keywords) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['news:keywords'], item.news.keywords));
}
if (item.news.stock_tickers) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['news:stock_tickers'], item.news.stock_tickers));
}
this.push((0, sitemap_xml_js_1.ctag)(types_js_1.TagNames['news:news']));
}
// Image handling
item.img.forEach((image) => {
this.push((0, sitemap_xml_js_1.otag)(types_js_1.TagNames['image:image']));
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['image:loc'], image.url));
if (image.caption) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['image:caption'], image.caption));
}
if (image.geoLocation) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['image:geo_location'], image.geoLocation));
}
if (image.title) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['image:title'], image.title));
}
if (image.license) {
this.push((0, sitemap_xml_js_1.element)(types_js_1.TagNames['image:license'], image.license));
}
this.push((0, sitemap_xml_js_1.ctag)(types_js_1.TagNames['image:image']));
});
this.push((0, sitemap_xml_js_1.ctag)(types_js_1.TagNames.url));
callback();
}
}
exports.SitemapItemStream = SitemapItemStream;
node_modules/sitemap/dist/cjs/lib/sitemap-parser.d.ts
Generated Vendored
+62
View File
@@ -0,0 +1,62 @@
import type { SAXStream } from 'sax';
import { Readable, Transform, TransformOptions, TransformCallback } from 'node:stream';
import { SitemapItem, ErrorLevel } from './types.js';
type Logger = (level: 'warn' | 'error' | 'info' | 'log', ...message: Parameters<Console['log']>[0]) => void;
export interface XMLToSitemapItemStreamOptions extends TransformOptions {
level?: ErrorLevel;
logger?: Logger | false;
}
/**
* Takes a stream of xml and transforms it into a stream of SitemapItems
* Use this to parse existing sitemaps into config options compatible with this library
*/
export declare class XMLToSitemapItemStream extends Transform {
level: ErrorLevel;
logger: Logger;
/**
* Errors encountered during parsing, capped at LIMITS.MAX_PARSER_ERRORS entries
* to prevent memory DoS from malformed XML (BB-03).
* Use errorCount for the total number of errors regardless of the cap.
*/
errors: Error[];
/** Total number of errors seen, including those beyond the stored cap. */
errorCount: number;
saxStream: SAXStream;
urlCount: number;
constructor(opts?: XMLToSitemapItemStreamOptions);
_transform(data: string, encoding: string, callback: TransformCallback): void;
private err;
}
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
const { createReadStream } = require('fs')
const { parseSitemap, createSitemap } = require('sitemap')
parseSitemap(createReadStream('./example.xml')).then(
// produces the same xml
// you can, of course, more practically modify it or store it
(xmlConfig) => console.log(createSitemap(xmlConfig).toString()),
(err) => console.log(err)
)
```
@param {Readable} xml what to parse
@return {Promise<SitemapItem[]>} resolves with list of sitemap items that can be fed into a SitemapStream. Rejects with an Error object.
*/
export declare function parseSitemap(xml: Readable): Promise<SitemapItem[]>;
export interface ObjectStreamToJSONOptions extends TransformOptions {
lineSeparated: boolean;
}
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
export declare class ObjectStreamToJSON extends Transform {
lineSeparated: boolean;
firstWritten: boolean;
constructor(opts?: ObjectStreamToJSONOptions);
_transform(chunk: SitemapItem, encoding: string, cb: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
export {};
node_modules/sitemap/dist/cjs/lib/sitemap-parser.js
Generated Vendored
+788
View File
@@ -0,0 +1,788 @@
"use strict";
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
exports.ObjectStreamToJSON = exports.XMLToSitemapItemStream = void 0;
exports.parseSitemap = parseSitemap;
const sax_1 = __importDefault(require("sax"));
const node_stream_1 = require("node:stream");
const types_js_1 = require("./types.js");
const validation_js_1 = require("./validation.js");
const constants_js_1 = require("./constants.js");
function isValidTagName(tagName) {
// This only works because the enum name and value are the same
return tagName in types_js_1.TagNames;
}
function getAttrValue(attr) {
if (!attr)
return undefined;
return typeof attr === 'string' ? attr : attr.value;
}
function tagTemplate() {
return {
img: [],
video: [],
links: [],
url: '',
};
}
function videoTemplate() {
return {
tag: [],
thumbnail_loc: '',
title: '',
description: '',
};
}
const imageTemplate = {
url: '',
};
const linkTemplate = {
lang: '',
url: '',
};
function newsTemplate() {
return {
publication: { name: '', language: '' },
publication_date: '',
title: '',
};
}
const defaultLogger = (level, ...message) => console[level](...message);
const defaultStreamOpts = {
logger: defaultLogger,
};
// TODO does this need to end with `options`
/**
* Takes a stream of xml and transforms it into a stream of SitemapItems
* Use this to parse existing sitemaps into config options compatible with this library
*/
class XMLToSitemapItemStream extends node_stream_1.Transform {
level;
logger;
/**
* Errors encountered during parsing, capped at LIMITS.MAX_PARSER_ERRORS entries
* to prevent memory DoS from malformed XML (BB-03).
* Use errorCount for the total number of errors regardless of the cap.
*/
errors;
/** Total number of errors seen, including those beyond the stored cap. */
errorCount;
saxStream;
urlCount;
constructor(opts = defaultStreamOpts) {
opts.objectMode = true;
super(opts);
this.errors = [];
this.errorCount = 0;
this.urlCount = 0;
this.saxStream = sax_1.default.createStream(true, {
xmlns: true,
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
strictEntities: true,
trim: true,
});
this.level = opts.level || types_js_1.ErrorLevel.WARN;
if (this.level !== types_js_1.ErrorLevel.SILENT && opts.logger !== false) {
this.logger = opts.logger ?? defaultLogger;
}
else {
this.logger = () => undefined;
}
let currentItem = tagTemplate();
let currentTag;
let currentVideo = videoTemplate();
let currentImage = { ...imageTemplate };
let currentLink = { ...linkTemplate };
let dontpushCurrentLink = false;
this.saxStream.on('opentagstart', (tag) => {
currentTag = tag.name;
if (currentTag.startsWith('news:') && !currentItem.news) {
currentItem.news = newsTemplate();
}
});
this.saxStream.on('opentag', (tag) => {
if (isValidTagName(tag.name)) {
if (tag.name === 'xhtml:link') {
// SAX returns attributes as objects with {name, value, prefix, local, uri}
// Check if required attributes exist and have values
const rel = getAttrValue(tag.attributes.rel);
const href = getAttrValue(tag.attributes.href);
const hreflang = getAttrValue(tag.attributes.hreflang);
if (!rel || !href) {
this.logger('warn', 'xhtml:link missing required rel or href attribute');
this.err('xhtml:link missing required rel or href attribute');
return;
}
if (rel === 'alternate' && hreflang) {
currentLink.url = href;
currentLink.lang = hreflang;
}
else if (rel === 'alternate') {
dontpushCurrentLink = true;
currentItem.androidLink = href;
}
else if (rel === 'amphtml') {
dontpushCurrentLink = true;
currentItem.ampLink = href;
}
else {
this.logger('log', 'unhandled attr for xhtml:link', tag.attributes);
this.err(`unhandled attr for xhtml:link ${JSON.stringify(tag.attributes)}`);
}
}
}
else {
this.logger('warn', 'unhandled tag', tag.name);
this.err(`unhandled tag: ${tag.name}`);
}
});
this.saxStream.on('text', (text) => {
switch (currentTag) {
case 'mobile:mobile':
break;
case types_js_1.TagNames.loc:
// Validate URL
if (text.length > constants_js_1.LIMITS.MAX_URL_LENGTH) {
this.logger('warn', `URL exceeds max length of ${constants_js_1.LIMITS.MAX_URL_LENGTH}: ${text.substring(0, 100)}...`);
this.err(`URL exceeds max length of ${constants_js_1.LIMITS.MAX_URL_LENGTH}`);
}
else if (!constants_js_1.LIMITS.URL_PROTOCOL_REGEX.test(text)) {
this.logger('warn', `URL must start with http:// or https://: ${text}`);
this.err(`URL must start with http:// or https://: ${text}`);
}
else {
currentItem.url = text;
}
break;
case types_js_1.TagNames.changefreq:
if ((0, validation_js_1.isValidChangeFreq)(text)) {
currentItem.changefreq = text;
}
break;
case types_js_1.TagNames.priority:
{
const priority = parseFloat(text);
if (isNaN(priority) ||
!isFinite(priority) ||
priority < 0 ||
priority > 1) {
this.logger('warn', `Invalid priority "${text}" - must be between 0 and 1`);
this.err(`Invalid priority "${text}" - must be between 0 and 1`);
}
else {
currentItem.priority = priority;
}
}
break;
case types_js_1.TagNames.lastmod:
if (constants_js_1.LIMITS.ISO_DATE_REGEX.test(text)) {
currentItem.lastmod = text;
}
else {
this.logger('warn', `Invalid lastmod date format "${text}" - expected ISO 8601 format`);
this.err(`Invalid lastmod date format "${text}" - expected ISO 8601 format`);
}
break;
case types_js_1.TagNames['video:thumbnail_loc']:
currentVideo.thumbnail_loc = text;
break;
case types_js_1.TagNames['video:tag']:
if (currentVideo.tag.length < constants_js_1.LIMITS.MAX_TAGS_PER_VIDEO) {
currentVideo.tag.push(text);
}
else {
this.logger('warn', `video has too many tags (max ${constants_js_1.LIMITS.MAX_TAGS_PER_VIDEO})`);
this.err(`video has too many tags (max ${constants_js_1.LIMITS.MAX_TAGS_PER_VIDEO})`);
}
break;
case types_js_1.TagNames['video:duration']:
{
const duration = parseInt(text, 10);
if (isNaN(duration) ||
!isFinite(duration) ||
duration < 0 ||
duration > 28800) {
this.logger('warn', `Invalid video duration "${text}" - must be between 0 and 28800 seconds`);
this.err(`Invalid video duration "${text}" - must be between 0 and 28800 seconds`);
}
else {
currentVideo.duration = duration;
}
}
break;
case types_js_1.TagNames['video:player_loc']:
currentVideo.player_loc = text;
break;
case types_js_1.TagNames['video:content_loc']:
currentVideo.content_loc = text;
break;
case types_js_1.TagNames['video:requires_subscription']:
if ((0, validation_js_1.isValidYesNo)(text)) {
currentVideo.requires_subscription = text;
}
break;
case types_js_1.TagNames['video:publication_date']:
if (constants_js_1.LIMITS.ISO_DATE_REGEX.test(text)) {
currentVideo.publication_date = text;
}
else {
this.logger('warn', `Invalid video publication_date format "${text}" - expected ISO 8601 format`);
this.err(`Invalid video publication_date format "${text}" - expected ISO 8601 format`);
}
break;
case types_js_1.TagNames['video:id']:
currentVideo.id = text;
break;
case types_js_1.TagNames['video:restriction']:
currentVideo.restriction = text;
break;
case types_js_1.TagNames['video:view_count']:
{
const viewCount = parseInt(text, 10);
if (isNaN(viewCount) || !isFinite(viewCount) || viewCount < 0) {
this.logger('warn', `Invalid video view_count "${text}" - must be a positive integer`);
this.err(`Invalid video view_count "${text}" - must be a positive integer`);
}
else {
currentVideo.view_count = viewCount;
}
}
break;
case types_js_1.TagNames['video:uploader']:
currentVideo.uploader = text;
break;
case types_js_1.TagNames['video:family_friendly']:
if ((0, validation_js_1.isValidYesNo)(text)) {
currentVideo.family_friendly = text;
}
break;
case types_js_1.TagNames['video:expiration_date']:
if (constants_js_1.LIMITS.ISO_DATE_REGEX.test(text)) {
currentVideo.expiration_date = text;
}
else {
this.logger('warn', `Invalid video expiration_date format "${text}" - expected ISO 8601 format`);
this.err(`Invalid video expiration_date format "${text}" - expected ISO 8601 format`);
}
break;
case types_js_1.TagNames['video:platform']:
currentVideo.platform = text;
break;
case types_js_1.TagNames['video:price']:
currentVideo.price = text;
break;
case types_js_1.TagNames['video:rating']:
{
const rating = parseFloat(text);
if (isNaN(rating) ||
!isFinite(rating) ||
rating < 0 ||
rating > 5) {
this.logger('warn', `Invalid video rating "${text}" - must be between 0 and 5`);
this.err(`Invalid video rating "${text}" - must be between 0 and 5`);
}
else {
currentVideo.rating = rating;
}
}
break;
case types_js_1.TagNames['video:category']:
currentVideo.category = text;
break;
case types_js_1.TagNames['video:live']:
if ((0, validation_js_1.isValidYesNo)(text)) {
currentVideo.live = text;
}
break;
case types_js_1.TagNames['video:gallery_loc']:
currentVideo.gallery_loc = text;
break;
case types_js_1.TagNames['image:loc']:
currentImage.url = text;
break;
case types_js_1.TagNames['image:geo_location']:
currentImage.geoLocation = text;
break;
case types_js_1.TagNames['image:license']:
currentImage.license = text;
break;
case types_js_1.TagNames['news:access']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (text === 'Registration' || text === 'Subscription') {
currentItem.news.access = text;
}
else {
this.logger('warn', `Invalid news:access value "${text}" - must be "Registration" or "Subscription"`);
this.err(`Invalid news:access value "${text}" - must be "Registration" or "Subscription"`);
}
break;
case types_js_1.TagNames['news:genres']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.genres = text;
break;
case types_js_1.TagNames['news:publication_date']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (constants_js_1.LIMITS.ISO_DATE_REGEX.test(text)) {
currentItem.news.publication_date = text;
}
else {
this.logger('warn', `Invalid news publication_date format "${text}" - expected ISO 8601 format`);
this.err(`Invalid news publication_date format "${text}" - expected ISO 8601 format`);
}
break;
case types_js_1.TagNames['news:keywords']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.keywords = text;
break;
case types_js_1.TagNames['news:stock_tickers']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.stock_tickers = text;
break;
case types_js_1.TagNames['news:language']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.publication.language = text;
break;
case types_js_1.TagNames['video:title']:
if (currentVideo.title.length + text.length <=
constants_js_1.LIMITS.MAX_VIDEO_TITLE_LENGTH) {
currentVideo.title += text;
}
else {
this.logger('warn', `video title exceeds max length of ${constants_js_1.LIMITS.MAX_VIDEO_TITLE_LENGTH}`);
this.err(`video title exceeds max length of ${constants_js_1.LIMITS.MAX_VIDEO_TITLE_LENGTH}`);
}
break;
case types_js_1.TagNames['video:description']:
if (currentVideo.description.length + text.length <=
constants_js_1.LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH) {
currentVideo.description += text;
}
else {
this.logger('warn', `video description exceeds max length of ${constants_js_1.LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH}`);
this.err(`video description exceeds max length of ${constants_js_1.LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH}`);
}
break;
case types_js_1.TagNames['news:name']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (currentItem.news.publication.name.length + text.length <=
constants_js_1.LIMITS.MAX_NEWS_NAME_LENGTH) {
currentItem.news.publication.name += text;
}
else {
this.logger('warn', `news name exceeds max length of ${constants_js_1.LIMITS.MAX_NEWS_NAME_LENGTH}`);
this.err(`news name exceeds max length of ${constants_js_1.LIMITS.MAX_NEWS_NAME_LENGTH}`);
}
break;
case types_js_1.TagNames['news:title']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (currentItem.news.title.length + text.length <=
constants_js_1.LIMITS.MAX_NEWS_TITLE_LENGTH) {
currentItem.news.title += text;
}
else {
this.logger('warn', `news title exceeds max length of ${constants_js_1.LIMITS.MAX_NEWS_TITLE_LENGTH}`);
this.err(`news title exceeds max length of ${constants_js_1.LIMITS.MAX_NEWS_TITLE_LENGTH}`);
}
break;
case types_js_1.TagNames['image:caption']:
if (!currentImage.caption) {
currentImage.caption =
text.length <= constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH
? text
: text.substring(0, constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH);
if (text.length > constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH) {
this.logger('warn', `image caption exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
this.err(`image caption exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
}
}
else if (currentImage.caption.length + text.length <=
constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH) {
currentImage.caption += text;
}
else {
this.logger('warn', `image caption exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
this.err(`image caption exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
}
break;
case types_js_1.TagNames['image:title']:
if (!currentImage.title) {
currentImage.title =
text.length <= constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH
? text
: text.substring(0, constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH);
if (text.length > constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH) {
this.logger('warn', `image title exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
this.err(`image title exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
}
}
else if (currentImage.title.length + text.length <=
constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH) {
currentImage.title += text;
}
else {
this.logger('warn', `image title exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
this.err(`image title exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
}
break;
default:
this.logger('log', 'unhandled text for tag:', currentTag, `'${text}'`);
this.err(`unhandled text for tag: ${currentTag} '${text}'`);
break;
}
});
this.saxStream.on('cdata', (text) => {
switch (currentTag) {
case types_js_1.TagNames.loc:
// Validate URL
if (text.length > constants_js_1.LIMITS.MAX_URL_LENGTH) {
this.logger('warn', `URL exceeds max length of ${constants_js_1.LIMITS.MAX_URL_LENGTH}: ${text.substring(0, 100)}...`);
this.err(`URL exceeds max length of ${constants_js_1.LIMITS.MAX_URL_LENGTH}`);
}
else if (!constants_js_1.LIMITS.URL_PROTOCOL_REGEX.test(text)) {
this.logger('warn', `URL must start with http:// or https://: ${text}`);
this.err(`URL must start with http:// or https://: ${text}`);
}
else {
currentItem.url = text;
}
break;
case types_js_1.TagNames['image:loc']:
currentImage.url = text;
break;
case types_js_1.TagNames['video:title']:
if (currentVideo.title.length + text.length <=
constants_js_1.LIMITS.MAX_VIDEO_TITLE_LENGTH) {
currentVideo.title += text;
}
else {
this.logger('warn', `video title exceeds max length of ${constants_js_1.LIMITS.MAX_VIDEO_TITLE_LENGTH}`);
this.err(`video title exceeds max length of ${constants_js_1.LIMITS.MAX_VIDEO_TITLE_LENGTH}`);
}
break;
case types_js_1.TagNames['video:description']:
if (currentVideo.description.length + text.length <=
constants_js_1.LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH) {
currentVideo.description += text;
}
else {
this.logger('warn', `video description exceeds max length of ${constants_js_1.LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH}`);
this.err(`video description exceeds max length of ${constants_js_1.LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH}`);
}
break;
case types_js_1.TagNames['news:name']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (currentItem.news.publication.name.length + text.length <=
constants_js_1.LIMITS.MAX_NEWS_NAME_LENGTH) {
currentItem.news.publication.name += text;
}
else {
this.logger('warn', `news name exceeds max length of ${constants_js_1.LIMITS.MAX_NEWS_NAME_LENGTH}`);
this.err(`news name exceeds max length of ${constants_js_1.LIMITS.MAX_NEWS_NAME_LENGTH}`);
}
break;
case types_js_1.TagNames['news:title']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (currentItem.news.title.length + text.length <=
constants_js_1.LIMITS.MAX_NEWS_TITLE_LENGTH) {
currentItem.news.title += text;
}
else {
this.logger('warn', `news title exceeds max length of ${constants_js_1.LIMITS.MAX_NEWS_TITLE_LENGTH}`);
this.err(`news title exceeds max length of ${constants_js_1.LIMITS.MAX_NEWS_TITLE_LENGTH}`);
}
break;
case types_js_1.TagNames['image:caption']:
if (!currentImage.caption) {
currentImage.caption =
text.length <= constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH
? text
: text.substring(0, constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH);
if (text.length > constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH) {
this.logger('warn', `image caption exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
this.err(`image caption exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
}
}
else if (currentImage.caption.length + text.length <=
constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH) {
currentImage.caption += text;
}
else {
this.logger('warn', `image caption exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
this.err(`image caption exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
}
break;
case types_js_1.TagNames['image:title']:
if (!currentImage.title) {
currentImage.title =
text.length <= constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH
? text
: text.substring(0, constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH);
if (text.length > constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH) {
this.logger('warn', `image title exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
this.err(`image title exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
}
}
else if (currentImage.title.length + text.length <=
constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH) {
currentImage.title += text;
}
else {
this.logger('warn', `image title exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
this.err(`image title exceeds max length of ${constants_js_1.LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
}
break;
default:
this.logger('log', 'unhandled cdata for tag:', currentTag);
this.err(`unhandled cdata for tag: ${currentTag}`);
break;
}
});
this.saxStream.on('attribute', (attr) => {
switch (currentTag) {
case types_js_1.TagNames['urlset']:
case types_js_1.TagNames['xhtml:link']:
case types_js_1.TagNames['video:id']:
break;
case types_js_1.TagNames['video:restriction']:
if (attr.name === 'relationship' && (0, validation_js_1.isAllowDeny)(attr.value)) {
currentVideo['restriction:relationship'] = attr.value;
}
else {
this.logger('log', 'unhandled attr', currentTag, attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case types_js_1.TagNames['video:price']:
if (attr.name === 'type' && (0, validation_js_1.isPriceType)(attr.value)) {
currentVideo['price:type'] = attr.value;
}
else if (attr.name === 'currency') {
currentVideo['price:currency'] = attr.value;
}
else if (attr.name === 'resolution' && (0, validation_js_1.isResolution)(attr.value)) {
currentVideo['price:resolution'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:price', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case types_js_1.TagNames['video:player_loc']:
if (attr.name === 'autoplay') {
currentVideo['player_loc:autoplay'] = attr.value;
}
else if (attr.name === 'allow_embed' && (0, validation_js_1.isValidYesNo)(attr.value)) {
currentVideo['player_loc:allow_embed'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:player_loc', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case types_js_1.TagNames['video:platform']:
if (attr.name === 'relationship' && (0, validation_js_1.isAllowDeny)(attr.value)) {
currentVideo['platform:relationship'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:platform', attr.name, attr.value);
this.err(`unhandled attr: ${currentTag} ${attr.name} ${attr.value}`);
}
break;
case types_js_1.TagNames['video:gallery_loc']:
if (attr.name === 'title') {
currentVideo['gallery_loc:title'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:galler_loc', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case types_js_1.TagNames['video:uploader']:
if (attr.name === 'info') {
currentVideo['uploader:info'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:uploader', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
default:
this.logger('log', 'unhandled attr', currentTag, attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
});
this.saxStream.on('closetag', (tag) => {
switch (tag) {
case types_js_1.TagNames.url:
this.urlCount++;
if (this.urlCount > constants_js_1.LIMITS.MAX_URL_ENTRIES) {
this.logger('error', `Sitemap exceeds maximum of ${constants_js_1.LIMITS.MAX_URL_ENTRIES} URLs`);
this.err(`Sitemap exceeds maximum of ${constants_js_1.LIMITS.MAX_URL_ENTRIES} URLs`);
currentItem = tagTemplate();
break;
}
this.push(currentItem);
currentItem = tagTemplate();
break;
case types_js_1.TagNames['video:video']:
if (currentItem.video.length < constants_js_1.LIMITS.MAX_VIDEOS_PER_URL) {
currentItem.video.push(currentVideo);
}
else {
this.logger('warn', `URL has too many videos (max ${constants_js_1.LIMITS.MAX_VIDEOS_PER_URL})`);
this.err(`URL has too many videos (max ${constants_js_1.LIMITS.MAX_VIDEOS_PER_URL})`);
}
currentVideo = videoTemplate();
break;
case types_js_1.TagNames['image:image']:
if (currentItem.img.length < constants_js_1.LIMITS.MAX_IMAGES_PER_URL) {
currentItem.img.push(currentImage);
}
else {
this.logger('warn', `URL has too many images (max ${constants_js_1.LIMITS.MAX_IMAGES_PER_URL})`);
this.err(`URL has too many images (max ${constants_js_1.LIMITS.MAX_IMAGES_PER_URL})`);
}
currentImage = { ...imageTemplate };
break;
case types_js_1.TagNames['xhtml:link']:
if (!dontpushCurrentLink) {
if (currentItem.links.length < constants_js_1.LIMITS.MAX_LINKS_PER_URL) {
currentItem.links.push(currentLink);
}
else {
this.logger('warn', `URL has too many links (max ${constants_js_1.LIMITS.MAX_LINKS_PER_URL})`);
this.err(`URL has too many links (max ${constants_js_1.LIMITS.MAX_LINKS_PER_URL})`);
}
}
currentLink = { ...linkTemplate };
dontpushCurrentLink = false; // Reset flag for next link
break;
default:
break;
}
});
}
_transform(data, encoding, callback) {
try {
const cb = () => callback(this.level === types_js_1.ErrorLevel.THROW && this.errors.length > 0
? this.errors[0]
: null);
// correcting the type here can be done without making it a breaking change
// TODO fix this
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
if (!this.saxStream.write(data, encoding)) {
this.saxStream.once('drain', cb);
}
else {
process.nextTick(cb);
}
}
catch (error) {
callback(error);
}
}
err(msg) {
this.errorCount++;
if (this.errors.length < constants_js_1.LIMITS.MAX_PARSER_ERRORS) {
this.errors.push(new Error(msg));
}
}
}
exports.XMLToSitemapItemStream = XMLToSitemapItemStream;
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
const { createReadStream } = require('fs')
const { parseSitemap, createSitemap } = require('sitemap')
parseSitemap(createReadStream('./example.xml')).then(
// produces the same xml
// you can, of course, more practically modify it or store it
(xmlConfig) => console.log(createSitemap(xmlConfig).toString()),
(err) => console.log(err)
)
```
@param {Readable} xml what to parse
@return {Promise<SitemapItem[]>} resolves with list of sitemap items that can be fed into a SitemapStream. Rejects with an Error object.
*/
async function parseSitemap(xml) {
const urls = [];
return new Promise((resolve, reject) => {
xml
.pipe(new XMLToSitemapItemStream())
.on('data', (smi) => urls.push(smi))
.on('end', () => {
resolve(urls);
})
.on('error', (error) => {
reject(error);
});
});
}
const defaultObjectStreamOpts = {
lineSeparated: false,
};
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
class ObjectStreamToJSON extends node_stream_1.Transform {
lineSeparated;
firstWritten;
constructor(opts = defaultObjectStreamOpts) {
opts.writableObjectMode = true;
super(opts);
this.lineSeparated = opts.lineSeparated;
this.firstWritten = false;
}
_transform(chunk, encoding, cb) {
if (!this.firstWritten) {
this.firstWritten = true;
if (!this.lineSeparated) {
this.push('[');
}
}
else if (this.lineSeparated) {
this.push('\n');
}
else {
this.push(',');
}
if (chunk) {
this.push(JSON.stringify(chunk));
}
cb();
}
_flush(cb) {
if (!this.lineSeparated) {
this.push(']');
}
cb();
}
}
exports.ObjectStreamToJSON = ObjectStreamToJSON;
node_modules/sitemap/dist/cjs/lib/sitemap-simple.d.ts
Generated Vendored
+63
View File
@@ -0,0 +1,63 @@
import { Readable } from 'node:stream';
import { SitemapItemLoose } from './types.js';
/**
* Options for the simpleSitemapAndIndex function
*/
export interface SimpleSitemapAndIndexOptions {
/**
* The hostname for all URLs
* Must be a valid http:// or https:// URL
*/
hostname: string;
/**
* The hostname for the sitemaps if different than hostname
* Must be a valid http:// or https:// URL
*/
sitemapHostname?: string;
/**
* The urls you want to make a sitemap out of.
* Can be an array of items, a file path string, a Readable stream, or an array of strings
*/
sourceData: SitemapItemLoose[] | string | Readable | string[];
/**
* Where to write the sitemaps and index
* Must be a relative path without path traversal sequences
*/
destinationDir: string;
/**
* Where the sitemaps are relative to the hostname. Defaults to root.
* Must not contain path traversal sequences
*/
publicBasePath?: string;
/**
* How many URLs to write before switching to a new file
* Must be between 1 and 50,000 per sitemaps.org spec
* @default 50000
*/
limit?: number;
/**
* Whether to compress the written files
* @default true
*/
gzip?: boolean;
/**
* Optional URL to an XSL stylesheet
* Must be a valid http:// or https:// URL
*/
xslUrl?: string;
}
/**
* A simpler interface for creating sitemaps and indexes.
* Automatically handles splitting large datasets into multiple sitemap files.
*
* @param options - Configuration options
* @returns A promise that resolves when all sitemaps and the index are written
* @throws {InvalidHostnameError} If hostname or sitemapHostname is invalid
* @throws {InvalidPathError} If destinationDir contains path traversal
* @throws {InvalidPublicBasePathError} If publicBasePath is invalid
* @throws {InvalidLimitError} If limit is out of range
* @throws {InvalidXSLUrlError} If xslUrl is invalid
* @throws {Error} If sourceData type is not supported
*/
export declare const simpleSitemapAndIndex: ({ hostname, sitemapHostname, sourceData, destinationDir, limit, gzip, publicBasePath, xslUrl, }: SimpleSitemapAndIndexOptions) => Promise<void>;
export default simpleSitemapAndIndex;
node_modules/sitemap/dist/cjs/lib/sitemap-simple.js
Generated Vendored
+113
View File
@@ -0,0 +1,113 @@
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.simpleSitemapAndIndex = void 0;
const sitemap_index_stream_js_1 = require("./sitemap-index-stream.js");
const sitemap_stream_js_1 = require("./sitemap-stream.js");
const utils_js_1 = require("./utils.js");
const node_zlib_1 = require("node:zlib");
const node_fs_1 = require("node:fs");
const node_path_1 = require("node:path");
const node_stream_1 = require("node:stream");
const promises_1 = require("node:stream/promises");
const node_url_1 = require("node:url");
const validation_js_1 = require("./validation.js");
/**
* A simpler interface for creating sitemaps and indexes.
* Automatically handles splitting large datasets into multiple sitemap files.
*
* @param options - Configuration options
* @returns A promise that resolves when all sitemaps and the index are written
* @throws {InvalidHostnameError} If hostname or sitemapHostname is invalid
* @throws {InvalidPathError} If destinationDir contains path traversal
* @throws {InvalidPublicBasePathError} If publicBasePath is invalid
* @throws {InvalidLimitError} If limit is out of range
* @throws {InvalidXSLUrlError} If xslUrl is invalid
* @throws {Error} If sourceData type is not supported
*/
const simpleSitemapAndIndex = async ({ hostname, sitemapHostname = hostname, // if different
sourceData, destinationDir, limit = 50000, gzip = true, publicBasePath = './', xslUrl, }) => {
// Validate all inputs upfront
(0, validation_js_1.validateURL)(hostname, 'hostname');
(0, validation_js_1.validateURL)(sitemapHostname, 'sitemapHostname');
(0, validation_js_1.validatePath)(destinationDir, 'destinationDir');
(0, validation_js_1.validateLimit)(limit);
(0, validation_js_1.validatePublicBasePath)(publicBasePath);
if (xslUrl) {
(0, validation_js_1.validateXSLUrl)(xslUrl);
}
// Create destination directory with error context
try {
await node_fs_1.promises.mkdir(destinationDir, { recursive: true });
}
catch (err) {
throw new Error(`Failed to create destination directory "${destinationDir}": ${err instanceof Error ? err.message : String(err)}`);
}
// Normalize publicBasePath (don't mutate the parameter)
const normalizedPublicBasePath = publicBasePath.endsWith('/')
? publicBasePath
: publicBasePath + '/';
const sitemapAndIndexStream = new sitemap_index_stream_js_1.SitemapAndIndexStream({
limit,
getSitemapStream: (i) => {
const sitemapStream = new sitemap_stream_js_1.SitemapStream({
hostname,
xslUrl,
});
const path = `./sitemap-${i}.xml`;
const writePath = (0, node_path_1.resolve)(destinationDir, path + (gzip ? '.gz' : ''));
// Construct public path for the sitemap index
const publicPath = (0, node_path_1.normalize)(normalizedPublicBasePath + path);
// Construct the URL with proper error handling
let sitemapUrl;
try {
sitemapUrl = new node_url_1.URL(`${publicPath}${gzip ? '.gz' : ''}`, sitemapHostname).toString();
}
catch (err) {
throw new Error(`Failed to construct sitemap URL for index ${i}: ${err instanceof Error ? err.message : String(err)}`);
}
let writeStream;
if (gzip) {
writeStream = sitemapStream
.pipe((0, node_zlib_1.createGzip)()) // compress the output of the sitemap
.pipe((0, node_fs_1.createWriteStream)(writePath)); // write it to sitemap-NUMBER.xml
}
else {
writeStream = sitemapStream.pipe((0, node_fs_1.createWriteStream)(writePath)); // write it to sitemap-NUMBER.xml
}
return [sitemapUrl, sitemapStream, writeStream];
},
});
// Handle different sourceData types with proper error handling
let src;
if (typeof sourceData === 'string') {
try {
src = (0, utils_js_1.lineSeparatedURLsToSitemapOptions)((0, node_fs_1.createReadStream)(sourceData));
}
catch (err) {
throw new Error(`Failed to read sourceData file "${sourceData}": ${err instanceof Error ? err.message : String(err)}`);
}
}
else if (sourceData instanceof node_stream_1.Readable) {
src = sourceData;
}
else if (Array.isArray(sourceData)) {
src = node_stream_1.Readable.from(sourceData);
}
else {
throw new Error(`Invalid sourceData type: expected array, string (file path), or Readable stream, got ${typeof sourceData}`);
}
const writePath = (0, node_path_1.resolve)(destinationDir, `./sitemap-index.xml${gzip ? '.gz' : ''}`);
try {
if (gzip) {
return await (0, promises_1.pipeline)(src, sitemapAndIndexStream, (0, node_zlib_1.createGzip)(), (0, node_fs_1.createWriteStream)(writePath));
}
else {
return await (0, promises_1.pipeline)(src, sitemapAndIndexStream, (0, node_fs_1.createWriteStream)(writePath));
}
}
catch (err) {
throw new Error(`Failed to write sitemap files: ${err instanceof Error ? err.message : String(err)}`);
}
};
exports.simpleSitemapAndIndex = simpleSitemapAndIndex;
exports.default = exports.simpleSitemapAndIndex;
node_modules/sitemap/dist/cjs/lib/sitemap-stream.d.ts
Generated Vendored
+79
View File
@@ -0,0 +1,79 @@
import { Transform, TransformOptions, TransformCallback, Readable } from 'node:stream';
import { SitemapItemLoose, ErrorLevel, ErrorHandler } from './types.js';
export declare const stylesheetInclude: (url: string) => string;
export interface NSArgs {
news: boolean;
video: boolean;
xhtml: boolean;
image: boolean;
custom?: string[];
}
export declare const closetag = "</urlset>";
export interface SitemapStreamOptions extends TransformOptions {
hostname?: string;
level?: ErrorLevel;
lastmodDateOnly?: boolean;
xmlns?: NSArgs;
xslUrl?: string;
errorHandler?: ErrorHandler;
}
/**
* A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream)
* for turning a
* [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams)
* of either [SitemapItemOptions](#sitemap-item-options) or url strings into a
* Sitemap. The readable stream it transforms **must** be in object mode.
*
* @param {SitemapStreamOptions} opts - Configuration options
* @param {string} [opts.hostname] - Base URL for relative paths. Must use http:// or https:// protocol
* @param {ErrorLevel} [opts.level=ErrorLevel.WARN] - Error handling level (SILENT, WARN, or THROW)
* @param {boolean} [opts.lastmodDateOnly=false] - Format lastmod as date only (YYYY-MM-DD)
* @param {NSArgs} [opts.xmlns] - Control which XML namespaces to include in output
* @param {string} [opts.xslUrl] - URL to XSL stylesheet for sitemap display. Must use http:// or https://
* @param {ErrorHandler} [opts.errorHandler] - Custom error handler function
*
* @throws {InvalidHostnameError} If hostname is provided but invalid (non-http(s), malformed, or >2048 chars)
* @throws {InvalidXSLUrlError} If xslUrl is provided but invalid (non-http(s), malformed, >2048 chars, or contains malicious content)
* @throws {Error} If xmlns.custom contains invalid namespace declarations
*
* @example
* ```typescript
* const stream = new SitemapStream({
* hostname: 'https://example.com',
* level: ErrorLevel.THROW
* });
* stream.write({ url: '/page', changefreq: 'daily' });
* stream.end();
* ```
*
* @security
* - Hostname and xslUrl are validated to prevent URL injection attacks
* - Custom namespaces are validated to prevent XML injection
* - All URLs are normalized and validated before output
* - XML content is properly escaped to prevent injection
*/
export declare class SitemapStream extends Transform {
hostname?: string;
level: ErrorLevel;
hasHeadOutput: boolean;
xmlNS: NSArgs;
xslUrl?: string;
errorHandler?: ErrorHandler;
private smiStream;
lastmodDateOnly: boolean;
constructor(opts?: SitemapStreamOptions);
_transform(item: SitemapItemLoose, encoding: string, callback: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
/**
* Converts a readable stream into a promise that resolves with the concatenated data from the stream.
*
* The function listens for 'data' events from the stream, and when the stream ends, it resolves the promise with the concatenated data. If an error occurs while reading from the stream, the promise is rejected with the error.
*
* ⚠️ CAUTION: This function should not generally be used in production / when writing to files as it holds a copy of the entire file contents in memory until finished.
*
* @param {Readable} stream - The readable stream to convert to a promise.
* @returns {Promise<Buffer>} A promise that resolves with the concatenated data from the stream as a Buffer, or rejects with an error if one occurred while reading from the stream. If the stream is empty, the promise is rejected with an EmptyStream error.
* @throws {EmptyStream} If the stream is empty.
*/
export declare function streamToPromise(stream: Readable): Promise<Buffer>;
node_modules/sitemap/dist/cjs/lib/sitemap-stream.js
Generated Vendored
+218
View File
@@ -0,0 +1,218 @@
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.SitemapStream = exports.closetag = exports.stylesheetInclude = void 0;
exports.streamToPromise = streamToPromise;
const node_stream_1 = require("node:stream");
const types_js_1 = require("./types.js");
const utils_js_1 = require("./utils.js");
const validation_js_1 = require("./validation.js");
const sitemap_item_stream_js_1 = require("./sitemap-item-stream.js");
const errors_js_1 = require("./errors.js");
const constants_js_1 = require("./constants.js");
const xmlDec = '<?xml version="1.0" encoding="UTF-8"?>';
const stylesheetInclude = (url) => {
const safe = url
.replace(/&/g, '&amp;')
.replace(/"/g, '&quot;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
return `<?xml-stylesheet type="text/xsl" href="${safe}"?>`;
};
exports.stylesheetInclude = stylesheetInclude;
const urlsetTagStart = '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"';
/**
* Validates custom namespace declarations for security
* @param custom - Array of custom namespace declarations
* @throws {Error} If namespace format is invalid or contains malicious content
*/
function validateCustomNamespaces(custom) {
if (!Array.isArray(custom)) {
throw new Error('Custom namespaces must be an array');
}
// Limit number of custom namespaces to prevent DoS
if (custom.length > constants_js_1.LIMITS.MAX_CUSTOM_NAMESPACES) {
throw new Error(`Too many custom namespaces: ${custom.length} exceeds limit of ${constants_js_1.LIMITS.MAX_CUSTOM_NAMESPACES}`);
}
// Basic format validation for xmlns declarations and namespace-qualified attributes
// Supports both xmlns:prefix="uri" and prefix:attribute="value" (e.g., xsi:schemaLocation)
const xmlAttributePattern = /^[a-zA-Z_][\w.-]*:[a-zA-Z_][\w.-]*="[^"<>]*"$/;
for (const ns of custom) {
if (typeof ns !== 'string' || ns.length === 0) {
throw new Error('Custom namespace must be a non-empty string');
}
if (ns.length > constants_js_1.LIMITS.MAX_NAMESPACE_LENGTH) {
throw new Error(`Custom namespace exceeds maximum length of ${constants_js_1.LIMITS.MAX_NAMESPACE_LENGTH} characters: ${ns.substring(0, 50)}...`);
}
// Check for potentially malicious content BEFORE format check
// (format check will reject < and > but we want specific error message)
const lowerNs = ns.toLowerCase();
if (lowerNs.includes('<script') ||
lowerNs.includes('javascript:') ||
lowerNs.includes('data:text/html')) {
throw new Error(`Custom namespace contains potentially malicious content: ${ns.substring(0, 50)}`);
}
// Check format matches xmlns declaration or namespace-qualified attribute
if (!xmlAttributePattern.test(ns)) {
throw new Error(`Invalid namespace format (must be prefix:name="value", e.g., xmlns:prefix="uri" or xsi:schemaLocation="..."): ${ns.substring(0, 50)}`);
}
}
}
const getURLSetNs = ({ news, video, image, xhtml, custom }, xslURL) => {
let ns = xmlDec;
if (xslURL) {
ns += (0, exports.stylesheetInclude)(xslURL);
}
ns += urlsetTagStart;
if (news) {
ns += ' xmlns:news="http://www.google.com/schemas/sitemap-news/0.9"';
}
if (xhtml) {
ns += ' xmlns:xhtml="http://www.w3.org/1999/xhtml"';
}
if (image) {
ns += ' xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"';
}
if (video) {
ns += ' xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"';
}
if (custom) {
validateCustomNamespaces(custom);
ns += ' ' + custom.join(' ');
}
return ns + '>';
};
exports.closetag = '</urlset>';
const defaultXMLNS = {
news: true,
xhtml: true,
image: true,
video: true,
};
const defaultStreamOpts = {
xmlns: defaultXMLNS,
};
/**
* A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream)
* for turning a
* [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams)
* of either [SitemapItemOptions](#sitemap-item-options) or url strings into a
* Sitemap. The readable stream it transforms **must** be in object mode.
*
* @param {SitemapStreamOptions} opts - Configuration options
* @param {string} [opts.hostname] - Base URL for relative paths. Must use http:// or https:// protocol
* @param {ErrorLevel} [opts.level=ErrorLevel.WARN] - Error handling level (SILENT, WARN, or THROW)
* @param {boolean} [opts.lastmodDateOnly=false] - Format lastmod as date only (YYYY-MM-DD)
* @param {NSArgs} [opts.xmlns] - Control which XML namespaces to include in output
* @param {string} [opts.xslUrl] - URL to XSL stylesheet for sitemap display. Must use http:// or https://
* @param {ErrorHandler} [opts.errorHandler] - Custom error handler function
*
* @throws {InvalidHostnameError} If hostname is provided but invalid (non-http(s), malformed, or >2048 chars)
* @throws {InvalidXSLUrlError} If xslUrl is provided but invalid (non-http(s), malformed, >2048 chars, or contains malicious content)
* @throws {Error} If xmlns.custom contains invalid namespace declarations
*
* @example
* ```typescript
* const stream = new SitemapStream({
* hostname: 'https://example.com',
* level: ErrorLevel.THROW
* });
* stream.write({ url: '/page', changefreq: 'daily' });
* stream.end();
* ```
*
* @security
* - Hostname and xslUrl are validated to prevent URL injection attacks
* - Custom namespaces are validated to prevent XML injection
* - All URLs are normalized and validated before output
* - XML content is properly escaped to prevent injection
*/
class SitemapStream extends node_stream_1.Transform {
hostname;
level;
hasHeadOutput;
xmlNS;
xslUrl;
errorHandler;
smiStream;
lastmodDateOnly;
constructor(opts = defaultStreamOpts) {
opts.objectMode = true;
super(opts);
// Validate hostname if provided
if (opts.hostname !== undefined) {
(0, validation_js_1.validateURL)(opts.hostname, 'hostname');
}
// Validate xslUrl if provided
if (opts.xslUrl !== undefined) {
(0, validation_js_1.validateXSLUrl)(opts.xslUrl);
}
this.hasHeadOutput = false;
this.hostname = opts.hostname;
this.level = opts.level || types_js_1.ErrorLevel.WARN;
this.errorHandler = opts.errorHandler;
this.smiStream = new sitemap_item_stream_js_1.SitemapItemStream({ level: opts.level });
this.smiStream.on('data', (data) => this.push(data));
this.lastmodDateOnly = opts.lastmodDateOnly || false;
this.xmlNS = opts.xmlns || defaultXMLNS;
this.xslUrl = opts.xslUrl;
}
_transform(item, encoding, callback) {
if (!this.hasHeadOutput) {
this.hasHeadOutput = true;
this.push(getURLSetNs(this.xmlNS, this.xslUrl));
}
if (!this.smiStream.write((0, validation_js_1.validateSMIOptions)((0, utils_js_1.normalizeURL)(item, this.hostname, this.lastmodDateOnly), this.level, this.errorHandler))) {
this.smiStream.once('drain', callback);
}
else {
process.nextTick(callback);
}
}
_flush(cb) {
if (!this.hasHeadOutput) {
cb(new errors_js_1.EmptySitemap());
}
else {
this.push(exports.closetag);
cb();
}
}
}
exports.SitemapStream = SitemapStream;
/**
* Converts a readable stream into a promise that resolves with the concatenated data from the stream.
*
* The function listens for 'data' events from the stream, and when the stream ends, it resolves the promise with the concatenated data. If an error occurs while reading from the stream, the promise is rejected with the error.
*
* ⚠️ CAUTION: This function should not generally be used in production / when writing to files as it holds a copy of the entire file contents in memory until finished.
*
* @param {Readable} stream - The readable stream to convert to a promise.
* @returns {Promise<Buffer>} A promise that resolves with the concatenated data from the stream as a Buffer, or rejects with an error if one occurred while reading from the stream. If the stream is empty, the promise is rejected with an EmptyStream error.
* @throws {EmptyStream} If the stream is empty.
*/
function streamToPromise(stream) {
return new Promise((resolve, reject) => {
const drain = [];
stream
// Error propagation is not automatic
// Bubble up errors on the read stream
.on('error', reject)
.pipe(new node_stream_1.Writable({
write(chunk, enc, next) {
drain.push(chunk);
next();
},
}))
// This bubbles up errors when writing to the internal buffer
// This is unlikely to happen, but we have this for completeness
.on('error', reject)
.on('finish', () => {
if (!drain.length) {
reject(new errors_js_1.EmptyStream());
}
else {
resolve(Buffer.concat(drain));
}
});
});
}
node_modules/sitemap/dist/cjs/lib/sitemap-xml.d.ts
Generated Vendored
+107
View File
@@ -0,0 +1,107 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { TagNames, IndexTagNames, StringObj } from './types.js';
/**
* Escapes text content for safe inclusion in XML text nodes.
*
* **Security Model:**
* - Escapes `&` → `&amp;` (required to prevent entity interpretation)
* - Escapes `<` → `&lt;` (required to prevent tag injection)
* - Escapes `>` → `&gt;` (defense-in-depth, prevents CDATA injection)
* - Does NOT escape `"` or `'` (not required in text content, only in attributes)
* - Removes invalid XML Unicode characters per XML 1.0 spec
*
* **Why quotes aren't escaped:**
* In XML text content (between tags), quotes have no special meaning and don't
* need escaping. They only need escaping in attribute values, which is handled
* by the `otag()` function.
*
* @param txt - The text content to escape
* @returns XML-safe escaped text with invalid characters removed
* @throws {TypeError} If txt is not a string
*
* @example
* text('Hello & World'); // Returns: 'Hello &amp; World'
* text('5 < 10'); // Returns: '5 &lt; 10'
* text('Hello "World"'); // Returns: 'Hello "World"' (quotes OK in text)
*
* @see https://www.w3.org/TR/xml/#syntax
*/
export declare function text(txt: string): string;
/**
* Generates an opening XML tag with optional attributes.
*
* **Security Model:**
* - Validates attribute names to prevent injection via malformed names
* - Escapes all attribute values with proper XML entity encoding
* - Escapes `&`, `<`, `>`, `"`, and `'` in attribute values
* - Removes invalid XML Unicode characters
*
* Attribute values use full escaping (including quotes) because they appear
* within quoted strings in the XML output: `<tag attr="value">`.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @param attrs - Optional object mapping attribute names to string values
* @param selfClose - If true, generates a self-closing tag (e.g., `<tag/>`)
* @returns Opening XML tag string
* @throws {InvalidXMLAttributeNameError} If an attribute name contains invalid characters
* @throws {TypeError} If nodeName is not a string or attrs values are not strings
*
* @example
* otag('url'); // Returns: '<url>'
* otag('video:player_loc', { autoplay: 'ap=1' }); // Returns: '<video:player_loc autoplay="ap=1">'
* otag('image:image', {}, true); // Returns: '<image:image/>'
*
* @see https://www.w3.org/TR/xml/#NT-Attribute
*/
export declare function otag(nodeName: TagNames | IndexTagNames, attrs?: StringObj, selfClose?: boolean): string;
/**
* Generates a closing XML tag.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @returns Closing XML tag string
* @throws {TypeError} If nodeName is not a string
*
* @example
* ctag('url'); // Returns: '</url>'
* ctag('video:title'); // Returns: '</video:title>'
*/
export declare function ctag(nodeName: TagNames | IndexTagNames): string;
/**
* Generates a complete XML element with optional attributes and text content.
*
* This is a convenience function that combines `otag()`, `text()`, and `ctag()`.
* It supports three usage patterns via function overloading:
*
* 1. Element with text content: `element('loc', 'https://example.com')`
* 2. Element with attributes and text: `element('video:player_loc', { autoplay: 'ap=1' }, 'https://...')`
* 3. Self-closing element with attributes: `element('image:image', { href: '...' })`
*
* @param nodeName - The XML element name
* @param attrs - Either a string (text content) or object (attributes)
* @param innerText - Optional text content when attrs is an object
* @returns Complete XML element string
* @throws {InvalidXMLAttributeNameError} If an attribute name contains invalid characters
* @throws {TypeError} If arguments have invalid types
*
* @example
* // Pattern 1: Simple element with text
* element('loc', 'https://example.com')
* // Returns: '<loc>https://example.com</loc>'
*
* @example
* // Pattern 2: Element with attributes and text
* element('video:player_loc', { autoplay: 'ap=1' }, 'https://example.com/video')
* // Returns: '<video:player_loc autoplay="ap=1">https://example.com/video</video:player_loc>'
*
* @example
* // Pattern 3: Self-closing element with attributes
* element('xhtml:link', { rel: 'alternate', href: 'https://example.com/fr' })
* // Returns: '<xhtml:link rel="alternate" href="https://example.com/fr"/>'
*/
export declare function element(nodeName: TagNames, attrs: StringObj, innerText: string): string;
export declare function element(nodeName: TagNames | IndexTagNames, innerText: string): string;
export declare function element(nodeName: TagNames, attrs: StringObj): string;
node_modules/sitemap/dist/cjs/lib/sitemap-xml.js
Generated Vendored
+187
View File
@@ -0,0 +1,187 @@
"use strict";
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
Object.defineProperty(exports, "__esModule", { value: true });
exports.text = text;
exports.otag = otag;
exports.ctag = ctag;
exports.element = element;
const errors_js_1 = require("./errors.js");
/**
* Regular expression matching invalid XML 1.0 Unicode characters that must be removed.
*
* Based on the XML 1.0 specification (https://www.w3.org/TR/xml/#charsets):
* - Control characters (U+0000-U+001F except tab, newline, carriage return)
* - Delete character (U+007F)
* - Invalid control characters (U+0080-U+009F except U+0085)
* - Surrogate pairs (U+D800-U+DFFF)
* - Non-characters (\p{NChar} - permanently reserved code points)
*
* Performance note: This regex uses Unicode property escapes and may be slower
* on very large strings (100KB+). Consider pre-validation for untrusted input.
*
* @see https://www.w3.org/TR/xml/#charsets
*/
const invalidXMLUnicodeRegex =
// eslint-disable-next-line no-control-regex
/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F-\u0084\u0086-\u009F\uD800-\uDFFF\p{NChar}]/gu;
/**
* Regular expressions for XML entity escaping
*/
const amp = /&/g;
const lt = /</g;
const gt = />/g;
const apos = /'/g;
const quot = /"/g;
/**
* Valid XML attribute name pattern. XML names must:
* - Start with a letter, underscore, or colon
* - Contain only letters, digits, hyphens, underscores, colons, or periods
*
* This is a simplified validation that accepts the most common attribute names.
* Note: In practice, this library only uses namespaced attributes like "video:title"
* which are guaranteed to be valid.
*
* @see https://www.w3.org/TR/xml/#NT-Name
*/
const validAttributeNameRegex = /^[a-zA-Z_:][\w:.-]*$/;
/**
* Validates that an attribute name is a valid XML identifier.
*
* XML attribute names must start with a letter, underscore, or colon,
* and contain only alphanumeric characters, hyphens, underscores, colons, or periods.
*
* @param name - The attribute name to validate
* @throws {InvalidXMLAttributeNameError} If the attribute name is invalid
*
* @example
* validateAttributeName('href'); // OK
* validateAttributeName('xml:lang'); // OK
* validateAttributeName('data-value'); // OK
* validateAttributeName('<script>'); // Throws InvalidXMLAttributeNameError
*/
function validateAttributeName(name) {
if (!validAttributeNameRegex.test(name)) {
throw new errors_js_1.InvalidXMLAttributeNameError(name);
}
}
/**
* Escapes text content for safe inclusion in XML text nodes.
*
* **Security Model:**
* - Escapes `&` → `&amp;` (required to prevent entity interpretation)
* - Escapes `<` → `&lt;` (required to prevent tag injection)
* - Escapes `>` → `&gt;` (defense-in-depth, prevents CDATA injection)
* - Does NOT escape `"` or `'` (not required in text content, only in attributes)
* - Removes invalid XML Unicode characters per XML 1.0 spec
*
* **Why quotes aren't escaped:**
* In XML text content (between tags), quotes have no special meaning and don't
* need escaping. They only need escaping in attribute values, which is handled
* by the `otag()` function.
*
* @param txt - The text content to escape
* @returns XML-safe escaped text with invalid characters removed
* @throws {TypeError} If txt is not a string
*
* @example
* text('Hello & World'); // Returns: 'Hello &amp; World'
* text('5 < 10'); // Returns: '5 &lt; 10'
* text('Hello "World"'); // Returns: 'Hello "World"' (quotes OK in text)
*
* @see https://www.w3.org/TR/xml/#syntax
*/
function text(txt) {
if (typeof txt !== 'string') {
throw new TypeError(`text() requires a string, received ${typeof txt}: ${String(txt)}`);
}
return txt
.replace(amp, '&amp;')
.replace(lt, '&lt;')
.replace(gt, '&gt;')
.replace(invalidXMLUnicodeRegex, '');
}
/**
* Generates an opening XML tag with optional attributes.
*
* **Security Model:**
* - Validates attribute names to prevent injection via malformed names
* - Escapes all attribute values with proper XML entity encoding
* - Escapes `&`, `<`, `>`, `"`, and `'` in attribute values
* - Removes invalid XML Unicode characters
*
* Attribute values use full escaping (including quotes) because they appear
* within quoted strings in the XML output: `<tag attr="value">`.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @param attrs - Optional object mapping attribute names to string values
* @param selfClose - If true, generates a self-closing tag (e.g., `<tag/>`)
* @returns Opening XML tag string
* @throws {InvalidXMLAttributeNameError} If an attribute name contains invalid characters
* @throws {TypeError} If nodeName is not a string or attrs values are not strings
*
* @example
* otag('url'); // Returns: '<url>'
* otag('video:player_loc', { autoplay: 'ap=1' }); // Returns: '<video:player_loc autoplay="ap=1">'
* otag('image:image', {}, true); // Returns: '<image:image/>'
*
* @see https://www.w3.org/TR/xml/#NT-Attribute
*/
function otag(nodeName, attrs, selfClose = false) {
if (typeof nodeName !== 'string') {
throw new TypeError(`otag() nodeName must be a string, received ${typeof nodeName}: ${String(nodeName)}`);
}
let attrstr = '';
for (const k in attrs) {
// Validate attribute name to prevent injection
validateAttributeName(k);
const attrValue = attrs[k];
if (typeof attrValue !== 'string') {
throw new TypeError(`otag() attribute "${k}" value must be a string, received ${typeof attrValue}: ${String(attrValue)}`);
}
// Escape attribute value with full entity encoding
const val = attrValue
.replace(amp, '&amp;')
.replace(lt, '&lt;')
.replace(gt, '&gt;')
.replace(apos, '&apos;')
.replace(quot, '&quot;')
.replace(invalidXMLUnicodeRegex, '');
attrstr += ` ${k}="${val}"`;
}
return `<${nodeName}${attrstr}${selfClose ? '/' : ''}>`;
}
/**
* Generates a closing XML tag.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @returns Closing XML tag string
* @throws {TypeError} If nodeName is not a string
*
* @example
* ctag('url'); // Returns: '</url>'
* ctag('video:title'); // Returns: '</video:title>'
*/
function ctag(nodeName) {
if (typeof nodeName !== 'string') {
throw new TypeError(`ctag() nodeName must be a string, received ${typeof nodeName}: ${String(nodeName)}`);
}
return `</${nodeName}>`;
}
function element(nodeName, attrs, innerText) {
if (typeof attrs === 'string') {
// Pattern 1: element(nodeName, textContent)
return otag(nodeName) + text(attrs) + ctag(nodeName);
}
else if (innerText !== undefined) {
// Pattern 2: element(nodeName, attrs, textContent)
return otag(nodeName, attrs) + text(innerText) + ctag(nodeName);
}
else {
// Pattern 3: element(nodeName, attrs) - self-closing
return otag(nodeName, attrs, true);
}
}
node_modules/sitemap/dist/cjs/lib/types.d.ts
Generated Vendored
+400
View File
@@ -0,0 +1,400 @@
import { URL } from 'node:url';
/**
* How frequently the page is likely to change. This value provides general
* information to search engines and may not correlate exactly to how often they crawl the page. Please note that the
* value of this tag is considered a hint and not a command. See
* <https://www.sitemaps.org/protocol.html#xmlTagDefinitions> for the acceptable
* values
*/
export declare enum EnumChangefreq {
DAILY = "daily",
MONTHLY = "monthly",
ALWAYS = "always",
HOURLY = "hourly",
WEEKLY = "weekly",
YEARLY = "yearly",
NEVER = "never"
}
export declare enum EnumYesNo {
YES = "YES",
NO = "NO",
Yes = "Yes",
No = "No",
yes = "yes",
no = "no"
}
export declare enum EnumAllowDeny {
ALLOW = "allow",
DENY = "deny"
}
/**
* https://support.google.com/webmasters/answer/74288?hl=en&ref_topic=4581190
*/
export interface NewsItem {
access?: 'Registration' | 'Subscription';
publication: {
name: string;
/**
* The `<language>` is the language of your publication. Use an ISO 639
* language code (2 or 3 letters).
*/
language: string;
};
/**
* @example 'PressRelease, Blog'
*/
genres?: string;
/**
* Article publication date in W3C format, using either the "complete date" (YYYY-MM-DD) format or the "complete date
* plus hours, minutes, and seconds"
*/
publication_date: string;
/**
* The title of the news article
* @example 'Companies A, B in Merger Talks'
*/
title: string;
/**
* @example 'business, merger, acquisition'
*/
keywords?: string;
/**
* @example 'NASDAQ:A, NASDAQ:B'
*/
stock_tickers?: string;
}
/**
* Sitemap Image
* https://support.google.com/webmasters/answer/178636?hl=en&ref_topic=4581190
*/
export interface Img {
/**
* The URL of the image
* @example 'https://example.com/image.jpg'
*/
url: string;
/**
* The caption of the image
* @example 'Thanksgiving dinner'
*/
caption?: string;
/**
* The title of the image
* @example 'Star Wars EP IV'
*/
title?: string;
/**
* The geographic location of the image.
* @example 'Limerick, Ireland'
*/
geoLocation?: string;
/**
* A URL to the license of the image.
* @example 'https://example.com/license.txt'
*/
license?: string;
}
interface VideoItemBase {
/**
* A URL pointing to the video thumbnail image file
* @example "https://rtv3-img-roosterteeth.akamaized.net/store/0e841100-289b-4184-ae30-b6a16736960a.jpg/sm/thumb3.jpg"
*/
thumbnail_loc: string;
/**
* The title of the video
* @example '2018:E6 - GoldenEye: Source'
*/
title: string;
/**
* A description of the video. Maximum 2048 characters.
* @example 'We play gun game in GoldenEye: Source with a good friend of ours. His name is Gruchy. Dan Gruchy.'
*/
description: string;
/**
* A URL pointing to the actual video media file. Should be one of the supported formats. HTML is not a supported
* format. Flash is allowed, but no longer supported on most mobile platforms, and so may be indexed less well. Must
* not be the same as the `<loc>` URL.
* @example "http://streamserver.example.com/video123.mp4"
*/
content_loc?: string;
/**
* A URL pointing to a player for a specific video. Usually this is the information in the src element of an `<embed>`
* tag. Must not be the same as the `<loc>` URL
* @example "https://roosterteeth.com/embed/rouletsplay-2018-goldeneye-source"
*/
player_loc?: string;
/**
* A string the search engine can append as a query param to enable automatic
* playback. Equivilant to auto play attr on player_loc tag.
* @example 'ap=1'
*/
'player_loc:autoplay'?: string;
/**
* Whether the search engine can embed the video in search results. Allowed values are yes or no.
*/
'player_loc:allow_embed'?: EnumYesNo;
/**
* The length of the video in seconds
* @example 600
*/
duration?: number;
/**
* The date after which the video will no longer be available.
* @example "2012-07-16T19:20:30+08:00"
*/
expiration_date?: string;
/**
* The number of times the video has been viewed
*/
view_count?: number;
/**
* The date the video was first published, in W3C format.
* @example "2012-07-16T19:20:30+08:00"
*/
publication_date?: string;
/**
* A short description of the broad category that the video belongs to. This is a string no longer than 256 characters.
* @example Baking
*/
category?: string;
/**
* Whether to show or hide your video in search results from specific countries.
* @example "IE GB US CA"
*/
restriction?: string;
/**
* Whether the countries in restriction are allowed or denied
* @example 'deny'
*/
'restriction:relationship'?: EnumAllowDeny;
gallery_loc?: string;
/**
* [Optional] Specifies the URL of a webpage with additional information about this uploader. This URL must be in the same domain as the <loc> tag.
* @see https://developers.google.com/search/docs/advanced/sitemaps/video-sitemaps
* @example http://www.example.com/users/grillymcgrillerson
*/
'uploader:info'?: string;
'gallery_loc:title'?: string;
/**
* The price to download or view the video. Omit this tag for free videos.
* @example "1.99"
*/
price?: string;
/**
* Specifies the resolution of the purchased version. Supported values are hd and sd.
* @example "HD"
*/
'price:resolution'?: Resolution;
/**
* Specifies the currency in ISO4217 format.
* @example "USD"
*/
'price:currency'?: string;
/**
* Specifies the purchase option. Supported values are rend and own.
* @example "rent"
*/
'price:type'?: PriceType;
/**
* The video uploader's name. Only one <video:uploader> is allowed per video. String value, max 255 characters.
* @example "GrillyMcGrillerson"
*/
uploader?: string;
/**
* Whether to show or hide your video in search results on specified platform types. This is a list of space-delimited
* platform types. See <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190> for more detail
* @example "tv"
*/
platform?: string;
id?: string;
'platform:relationship'?: EnumAllowDeny;
}
/**
* Video price type - supports both lowercase and uppercase variants
* as allowed by the Google Video Sitemap specification
* @see https://developers.google.com/search/docs/advanced/sitemaps/video-sitemaps
*/
export type PriceType = 'rent' | 'purchase' | 'RENT' | 'PURCHASE';
/**
* Video resolution - supports both lowercase and uppercase variants
* as allowed by the Google Video Sitemap specification
* @see https://developers.google.com/search/docs/advanced/sitemaps/video-sitemaps
*/
export type Resolution = 'HD' | 'hd' | 'sd' | 'SD';
/**
* Sitemap video. <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190>
*/
export interface VideoItem extends VideoItemBase {
/**
* An arbitrary string tag describing the video. Tags are generally very short descriptions of key concepts associated
* with a video or piece of content.
* @example ['Baking']
*/
tag: string[];
/**
* The rating of the video. Supported values are float numbers.
* @example 2.5
*/
rating?: number;
family_friendly?: EnumYesNo;
/**
* Indicates whether a subscription (either paid or free) is required to view
* the video. Allowed values are yes or no.
*/
requires_subscription?: EnumYesNo;
/**
* Indicates whether the video is a live stream. Supported values are yes or no.
*/
live?: EnumYesNo;
}
/**
* Sitemap video. <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190>
*/
export interface VideoItemLoose extends VideoItemBase {
/**
* An arbitrary string tag describing the video. Tags are generally very short descriptions of key concepts associated
* with a video or piece of content.
* @example ['Baking']
*/
tag?: string | string[];
/**
* The rating of the video. Supported values are float numbers.
* @example 2.5
*/
rating?: string | number;
family_friendly?: EnumYesNo | boolean;
requires_subscription?: EnumYesNo | boolean;
/**
* Indicates whether the video is a live stream. Supported values are yes or no.
*/
live?: EnumYesNo | boolean;
}
/**
* https://support.google.com/webmasters/answer/189077
*/
export interface LinkItem {
/**
* @example 'en'
*/
lang: string;
/**
* @example 'en-us'
*/
hreflang?: string;
url: string;
}
export interface IndexItem {
url: string;
lastmod?: string;
}
interface SitemapItemBase {
lastmod?: string;
changefreq?: EnumChangefreq;
fullPrecisionPriority?: boolean;
priority?: number;
news?: NewsItem;
expires?: string;
androidLink?: string;
ampLink?: string;
url: string;
}
/**
* Strict options for individual sitemap entries
*/
export interface SitemapItem extends SitemapItemBase {
img: Img[];
video: VideoItem[];
links: LinkItem[];
}
/**
* Options for individual sitemap entries prior to normalization
*/
export interface SitemapItemLoose extends SitemapItemBase {
video?: VideoItemLoose | VideoItemLoose[];
img?: string | Img | (string | Img)[];
links?: LinkItem[];
lastmodfile?: string | Buffer | URL;
lastmodISO?: string;
lastmodrealtime?: boolean;
}
/**
* How to handle errors in passed in urls
*/
export declare enum ErrorLevel {
/**
* Validation will be skipped and nothing logged or thrown.
*/
SILENT = "silent",
/**
* If an invalid value is encountered, a console.warn will be called with details
*/
WARN = "warn",
/**
* An Error will be thrown on encountering invalid data.
*/
THROW = "throw"
}
export type ErrorHandler = (error: Error, level: ErrorLevel) => void;
export declare enum TagNames {
url = "url",
loc = "loc",
urlset = "urlset",
lastmod = "lastmod",
changefreq = "changefreq",
priority = "priority",
'video:thumbnail_loc' = "video:thumbnail_loc",
'video:video' = "video:video",
'video:title' = "video:title",
'video:description' = "video:description",
'video:tag' = "video:tag",
'video:duration' = "video:duration",
'video:player_loc' = "video:player_loc",
'video:content_loc' = "video:content_loc",
'image:image' = "image:image",
'image:loc' = "image:loc",
'image:geo_location' = "image:geo_location",
'image:license' = "image:license",
'image:title' = "image:title",
'image:caption' = "image:caption",
'video:requires_subscription' = "video:requires_subscription",
'video:publication_date' = "video:publication_date",
'video:id' = "video:id",
'video:restriction' = "video:restriction",
'video:family_friendly' = "video:family_friendly",
'video:view_count' = "video:view_count",
'video:uploader' = "video:uploader",
'video:expiration_date' = "video:expiration_date",
'video:platform' = "video:platform",
'video:price' = "video:price",
'video:rating' = "video:rating",
'video:category' = "video:category",
'video:live' = "video:live",
'video:gallery_loc' = "video:gallery_loc",
'news:news' = "news:news",
'news:publication' = "news:publication",
'news:name' = "news:name",
'news:access' = "news:access",
'news:genres' = "news:genres",
'news:publication_date' = "news:publication_date",
'news:title' = "news:title",
'news:keywords' = "news:keywords",
'news:stock_tickers' = "news:stock_tickers",
'news:language' = "news:language",
'mobile:mobile' = "mobile:mobile",
'xhtml:link' = "xhtml:link",
'expires' = "expires"
}
export declare enum IndexTagNames {
sitemap = "sitemap",
sitemapindex = "sitemapindex",
loc = "loc",
lastmod = "lastmod"
}
/**
* Generic object with string keys and any values
* Used for XML attribute building and other flexible data structures
*/
export interface StringObj {
[index: string]: any;
}
export {};
node_modules/sitemap/dist/cjs/lib/types.js
Generated Vendored
+109
View File
@@ -0,0 +1,109 @@
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.IndexTagNames = exports.TagNames = exports.ErrorLevel = exports.EnumAllowDeny = exports.EnumYesNo = exports.EnumChangefreq = void 0;
/**
* How frequently the page is likely to change. This value provides general
* information to search engines and may not correlate exactly to how often they crawl the page. Please note that the
* value of this tag is considered a hint and not a command. See
* <https://www.sitemaps.org/protocol.html#xmlTagDefinitions> for the acceptable
* values
*/
var EnumChangefreq;
(function (EnumChangefreq) {
EnumChangefreq["DAILY"] = "daily";
EnumChangefreq["MONTHLY"] = "monthly";
EnumChangefreq["ALWAYS"] = "always";
EnumChangefreq["HOURLY"] = "hourly";
EnumChangefreq["WEEKLY"] = "weekly";
EnumChangefreq["YEARLY"] = "yearly";
EnumChangefreq["NEVER"] = "never";
})(EnumChangefreq || (exports.EnumChangefreq = EnumChangefreq = {}));
var EnumYesNo;
(function (EnumYesNo) {
EnumYesNo["YES"] = "YES";
EnumYesNo["NO"] = "NO";
EnumYesNo["Yes"] = "Yes";
EnumYesNo["No"] = "No";
EnumYesNo["yes"] = "yes";
EnumYesNo["no"] = "no";
})(EnumYesNo || (exports.EnumYesNo = EnumYesNo = {}));
var EnumAllowDeny;
(function (EnumAllowDeny) {
EnumAllowDeny["ALLOW"] = "allow";
EnumAllowDeny["DENY"] = "deny";
})(EnumAllowDeny || (exports.EnumAllowDeny = EnumAllowDeny = {}));
/**
* How to handle errors in passed in urls
*/
var ErrorLevel;
(function (ErrorLevel) {
/**
* Validation will be skipped and nothing logged or thrown.
*/
ErrorLevel["SILENT"] = "silent";
/**
* If an invalid value is encountered, a console.warn will be called with details
*/
ErrorLevel["WARN"] = "warn";
/**
* An Error will be thrown on encountering invalid data.
*/
ErrorLevel["THROW"] = "throw";
})(ErrorLevel || (exports.ErrorLevel = ErrorLevel = {}));
var TagNames;
(function (TagNames) {
TagNames["url"] = "url";
TagNames["loc"] = "loc";
TagNames["urlset"] = "urlset";
TagNames["lastmod"] = "lastmod";
TagNames["changefreq"] = "changefreq";
TagNames["priority"] = "priority";
TagNames["video:thumbnail_loc"] = "video:thumbnail_loc";
TagNames["video:video"] = "video:video";
TagNames["video:title"] = "video:title";
TagNames["video:description"] = "video:description";
TagNames["video:tag"] = "video:tag";
TagNames["video:duration"] = "video:duration";
TagNames["video:player_loc"] = "video:player_loc";
TagNames["video:content_loc"] = "video:content_loc";
TagNames["image:image"] = "image:image";
TagNames["image:loc"] = "image:loc";
TagNames["image:geo_location"] = "image:geo_location";
TagNames["image:license"] = "image:license";
TagNames["image:title"] = "image:title";
TagNames["image:caption"] = "image:caption";
TagNames["video:requires_subscription"] = "video:requires_subscription";
TagNames["video:publication_date"] = "video:publication_date";
TagNames["video:id"] = "video:id";
TagNames["video:restriction"] = "video:restriction";
TagNames["video:family_friendly"] = "video:family_friendly";
TagNames["video:view_count"] = "video:view_count";
TagNames["video:uploader"] = "video:uploader";
TagNames["video:expiration_date"] = "video:expiration_date";
TagNames["video:platform"] = "video:platform";
TagNames["video:price"] = "video:price";
TagNames["video:rating"] = "video:rating";
TagNames["video:category"] = "video:category";
TagNames["video:live"] = "video:live";
TagNames["video:gallery_loc"] = "video:gallery_loc";
TagNames["news:news"] = "news:news";
TagNames["news:publication"] = "news:publication";
TagNames["news:name"] = "news:name";
TagNames["news:access"] = "news:access";
TagNames["news:genres"] = "news:genres";
TagNames["news:publication_date"] = "news:publication_date";
TagNames["news:title"] = "news:title";
TagNames["news:keywords"] = "news:keywords";
TagNames["news:stock_tickers"] = "news:stock_tickers";
TagNames["news:language"] = "news:language";
TagNames["mobile:mobile"] = "mobile:mobile";
TagNames["xhtml:link"] = "xhtml:link";
TagNames["expires"] = "expires";
})(TagNames || (exports.TagNames = TagNames = {}));
var IndexTagNames;
(function (IndexTagNames) {
IndexTagNames["sitemap"] = "sitemap";
IndexTagNames["sitemapindex"] = "sitemapindex";
IndexTagNames["loc"] = "loc";
IndexTagNames["lastmod"] = "lastmod";
})(IndexTagNames || (exports.IndexTagNames = IndexTagNames = {}));
node_modules/sitemap/dist/cjs/lib/utils.d.ts
Generated Vendored
+48
View File
@@ -0,0 +1,48 @@
import { Readable, ReadableOptions, TransformOptions } from 'node:stream';
import { SitemapItem, SitemapItemLoose } from './types.js';
export { validateSMIOptions } from './validation.js';
/**
* Combines multiple streams into one
* @param streams the streams to combine
*/
export declare function mergeStreams(streams: Readable[], options?: TransformOptions): Readable;
export interface ReadlineStreamOptions extends ReadableOptions {
input: Readable;
}
/**
* Wraps node's ReadLine in a stream
*/
export declare class ReadlineStream extends Readable {
private _source;
constructor(options: ReadlineStreamOptions);
_read(size: number): void;
}
/**
* Takes a stream likely from fs.createReadStream('./path') and returns a stream
* of sitemap items
* @param stream a stream of line separated urls.
* @param opts.isJSON is the stream line separated JSON. leave undefined to guess
*/
export declare function lineSeparatedURLsToSitemapOptions(stream: Readable, { isJSON }?: {
isJSON?: boolean;
}): Readable;
/**
* Based on lodash's implementation of chunk.
*
* Copyright JS Foundation and other contributors <https://js.foundation/>
*
* Based on Underscore.js, copyright Jeremy Ashkenas,
* DocumentCloud and Investigative Reporters & Editors <http://underscorejs.org/>
*
* This software consists of voluntary contributions made by many
* individuals. For exact contribution history, see the revision history
* available at https://github.com/lodash/lodash
*/
export declare function chunk(array: any[], size?: number): any[];
/**
* Converts the passed in sitemap entry into one capable of being consumed by SitemapItem
* @param {string | SitemapItemLoose} elem the string or object to be converted
* @param {string} hostname
* @returns SitemapItemOptions a strict sitemap item option
*/
export declare function normalizeURL(elem: string | SitemapItemLoose, hostname?: string, lastmodDateOnly?: boolean): SitemapItem;
node_modules/sitemap/dist/cjs/lib/utils.js
Generated Vendored
+230
View File
@@ -0,0 +1,230 @@
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.ReadlineStream = exports.validateSMIOptions = void 0;
exports.mergeStreams = mergeStreams;
exports.lineSeparatedURLsToSitemapOptions = lineSeparatedURLsToSitemapOptions;
exports.chunk = chunk;
exports.normalizeURL = normalizeURL;
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
const node_fs_1 = require("node:fs");
const node_stream_1 = require("node:stream");
const node_readline_1 = require("node:readline");
const node_url_1 = require("node:url");
const types_js_1 = require("./types.js");
// Re-export validateSMIOptions from validation.ts for backward compatibility
var validation_js_1 = require("./validation.js");
Object.defineProperty(exports, "validateSMIOptions", { enumerable: true, get: function () { return validation_js_1.validateSMIOptions; } });
/**
* Combines multiple streams into one
* @param streams the streams to combine
*/
function mergeStreams(streams, options) {
let pass = new node_stream_1.PassThrough(options);
let waiting = streams.length;
for (const stream of streams) {
pass = stream.pipe(pass, { end: false });
stream.once('end', () => --waiting === 0 && pass.emit('end'));
}
return pass;
}
/**
* Wraps node's ReadLine in a stream
*/
class ReadlineStream extends node_stream_1.Readable {
_source;
constructor(options) {
if (options.autoDestroy === undefined) {
options.autoDestroy = true;
}
options.objectMode = true;
super(options);
this._source = (0, node_readline_1.createInterface)({
input: options.input,
terminal: false,
crlfDelay: Infinity,
});
// Every time there's data, push it into the internal buffer.
this._source.on('line', (chunk) => {
// If push() returns false, then stop reading from source.
if (!this.push(chunk))
this._source.pause();
});
// When the source ends, push the EOF-signaling `null` chunk.
this._source.on('close', () => {
this.push(null);
});
}
// _read() will be called when the stream wants to pull more data in.
// The advisory size argument is ignored in this case.
_read(size) {
this._source.resume();
}
}
exports.ReadlineStream = ReadlineStream;
/**
* Takes a stream likely from fs.createReadStream('./path') and returns a stream
* of sitemap items
* @param stream a stream of line separated urls.
* @param opts.isJSON is the stream line separated JSON. leave undefined to guess
*/
function lineSeparatedURLsToSitemapOptions(stream, { isJSON } = {}) {
return new ReadlineStream({ input: stream }).pipe(new node_stream_1.Transform({
objectMode: true,
transform: (line, encoding, cb) => {
if (isJSON || (isJSON === undefined && line[0] === '{')) {
cb(null, JSON.parse(line));
}
else {
cb(null, line);
}
},
}));
}
/**
* Based on lodash's implementation of chunk.
*
* Copyright JS Foundation and other contributors <https://js.foundation/>
*
* Based on Underscore.js, copyright Jeremy Ashkenas,
* DocumentCloud and Investigative Reporters & Editors <http://underscorejs.org/>
*
* This software consists of voluntary contributions made by many
* individuals. For exact contribution history, see the revision history
* available at https://github.com/lodash/lodash
*/
/* eslint-disable @typescript-eslint/no-explicit-any */
function chunk(array, size = 1) {
size = Math.max(Math.trunc(size), 0);
const length = array ? array.length : 0;
if (!length || size < 1) {
return [];
}
const result = Array(Math.ceil(length / size));
let index = 0, resIndex = 0;
while (index < length) {
result[resIndex++] = array.slice(index, (index += size));
}
return result;
}
function boolToYESNO(bool) {
if (bool === undefined) {
return undefined;
}
if (typeof bool === 'boolean') {
return bool ? types_js_1.EnumYesNo.yes : types_js_1.EnumYesNo.no;
}
return bool;
}
/**
* Converts the passed in sitemap entry into one capable of being consumed by SitemapItem
* @param {string | SitemapItemLoose} elem the string or object to be converted
* @param {string} hostname
* @returns SitemapItemOptions a strict sitemap item option
*/
function normalizeURL(elem, hostname, lastmodDateOnly = false) {
// SitemapItem
// create object with url property
const smi = {
img: [],
video: [],
links: [],
url: '',
};
if (typeof elem === 'string') {
smi.url = new node_url_1.URL(elem, hostname).toString();
return smi;
}
const { url, img, links, video, lastmodfile, lastmodISO, lastmod, ...other } = elem;
Object.assign(smi, other);
smi.url = new node_url_1.URL(url, hostname).toString();
if (img) {
// prepend hostname to all image urls
smi.img = (Array.isArray(img) ? img : [img]).map((el) => typeof el === 'string'
? { url: new node_url_1.URL(el, hostname).toString() }
: { ...el, url: new node_url_1.URL(el.url, hostname).toString() });
}
if (links) {
smi.links = links.map((link) => ({
...link,
url: new node_url_1.URL(link.url, hostname).toString(),
}));
}
if (video) {
smi.video = (Array.isArray(video) ? video : [video]).map((video) => {
const nv = {
...video,
family_friendly: boolToYESNO(video.family_friendly),
live: boolToYESNO(video.live),
requires_subscription: boolToYESNO(video.requires_subscription),
tag: [],
rating: undefined,
};
if (video.tag !== undefined) {
nv.tag = !Array.isArray(video.tag) ? [video.tag] : video.tag;
}
if (video.rating !== undefined) {
if (typeof video.rating === 'string') {
const parsedRating = parseFloat(video.rating);
// Validate parsed rating is a valid number
if (Number.isNaN(parsedRating)) {
throw new Error(`Invalid video rating "${video.rating}" for URL "${elem.url}": must be a valid number`);
}
nv.rating = parsedRating;
}
else {
nv.rating = video.rating;
}
}
if (typeof video.view_count === 'string') {
const parsedViewCount = parseInt(video.view_count, 10);
// Validate parsed view count is a valid non-negative integer
if (Number.isNaN(parsedViewCount)) {
throw new Error(`Invalid video view_count "${video.view_count}" for URL "${elem.url}": must be a valid number`);
}
if (parsedViewCount < 0) {
throw new Error(`Invalid video view_count "${video.view_count}" for URL "${elem.url}": cannot be negative`);
}
nv.view_count = parsedViewCount;
}
else if (typeof video.view_count === 'number') {
nv.view_count = video.view_count;
}
return nv;
});
}
// If given a file to use for last modified date
if (lastmodfile) {
const { mtime } = (0, node_fs_1.statSync)(lastmodfile);
const lastmodDate = new Date(mtime);
// Validate date is valid
if (Number.isNaN(lastmodDate.getTime())) {
throw new Error(`Invalid date from file stats for URL "${smi.url}": file modification time is invalid`);
}
smi.lastmod = lastmodDate.toISOString();
// The date of last modification (YYYY-MM-DD)
}
else if (lastmodISO) {
const lastmodDate = new Date(lastmodISO);
// Validate date is valid
if (Number.isNaN(lastmodDate.getTime())) {
throw new Error(`Invalid lastmodISO "${lastmodISO}" for URL "${smi.url}": must be a valid date string`);
}
smi.lastmod = lastmodDate.toISOString();
}
else if (lastmod) {
const lastmodDate = new Date(lastmod);
// Validate date is valid
if (Number.isNaN(lastmodDate.getTime())) {
throw new Error(`Invalid lastmod "${lastmod}" for URL "${smi.url}": must be a valid date string`);
}
smi.lastmod = lastmodDate.toISOString();
}
if (lastmodDateOnly && smi.lastmod) {
smi.lastmod = smi.lastmod.slice(0, 10);
}
return smi;
}
node_modules/sitemap/dist/cjs/lib/validation.d.ts
Generated Vendored
+94
View File
@@ -0,0 +1,94 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { SitemapItem, ErrorLevel, EnumChangefreq, EnumYesNo, EnumAllowDeny, PriceType, Resolution, ErrorHandler } from './types.js';
export declare const validators: {
[index: string]: RegExp;
};
/**
* Type guard to check if a string is a valid price type
*/
export declare function isPriceType(pt: string | PriceType): pt is PriceType;
/**
* Type guard to check if a string is a valid resolution
*/
export declare function isResolution(res: string): res is Resolution;
export declare function isValidChangeFreq(freq: string): freq is EnumChangefreq;
/**
* Type guard to check if a string is a valid yes/no value
*/
export declare function isValidYesNo(yn: string): yn is EnumYesNo;
/**
* Type guard to check if a string is a valid allow/deny value
*/
export declare function isAllowDeny(ad: string): ad is EnumAllowDeny;
/**
* Validates that a URL is well-formed and meets security requirements
*
* Security: This function enforces that URLs use safe protocols (http/https),
* are within reasonable length limits (2048 chars per sitemaps.org spec),
* and can be properly parsed. This prevents protocol injection attacks and
* ensures compliance with sitemap specifications.
*
* @param url - The URL to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidHostnameError} If the URL is invalid
*/
export declare function validateURL(url: string, paramName: string): void;
/**
* Validates that a path doesn't contain path traversal sequences
*
* Security: This function prevents path traversal attacks by detecting
* any occurrence of '..' in the path, whether it appears as '../', '/..',
* or standalone. This prevents attackers from accessing files outside
* the intended directory structure.
*
* @param path - The path to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidPathError} If the path contains traversal sequences
*/
export declare function validatePath(path: string, paramName: string): void;
/**
* Validates that a public base path is safe for URL construction
*
* Security: This function prevents path traversal attacks and validates
* that the path is safe for use in URL construction within sitemap indexes.
* It checks for '..' sequences, null bytes, and invalid whitespace that
* could be used to manipulate URL structure or inject malicious content.
*
* @param publicBasePath - The public base path to validate
* @throws {InvalidPublicBasePathError} If the path is invalid
*/
export declare function validatePublicBasePath(publicBasePath: string): void;
/**
* Validates that a limit is within acceptable range per sitemaps.org spec
*
* Security: This function enforces sitemap size limits (1-50,000 URLs per
* sitemap) as specified by sitemaps.org. This prevents resource exhaustion
* attacks and ensures compliance with search engine requirements.
*
* @param limit - The limit to validate
* @throws {InvalidLimitError} If the limit is out of range
*/
export declare function validateLimit(limit: number): void;
/**
* Validates that an XSL URL is safe and well-formed
*
* Security: This function validates XSL stylesheet URLs to prevent
* injection attacks. It blocks dangerous protocols and content patterns
* that could be used for XSS or other attacks. The validation uses
* case-insensitive matching to catch obfuscated attacks.
*
* @param xslUrl - The XSL URL to validate
* @throws {InvalidXSLUrlError} If the URL is invalid
*/
export declare function validateXSLUrl(xslUrl: string): void;
/**
* Verifies all data passed in will comply with sitemap spec.
* @param conf Options to validate
* @param level logging level
* @param errorHandler error handling func
*/
export declare function validateSMIOptions(conf: SitemapItem, level?: ErrorLevel, errorHandler?: ErrorHandler): SitemapItem;
node_modules/sitemap/dist/cjs/lib/validation.js
Generated Vendored
+398
View File
@@ -0,0 +1,398 @@
"use strict";
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
Object.defineProperty(exports, "__esModule", { value: true });
exports.validators = void 0;
exports.isPriceType = isPriceType;
exports.isResolution = isResolution;
exports.isValidChangeFreq = isValidChangeFreq;
exports.isValidYesNo = isValidYesNo;
exports.isAllowDeny = isAllowDeny;
exports.validateURL = validateURL;
exports.validatePath = validatePath;
exports.validatePublicBasePath = validatePublicBasePath;
exports.validateLimit = validateLimit;
exports.validateXSLUrl = validateXSLUrl;
exports.validateSMIOptions = validateSMIOptions;
const errors_js_1 = require("./errors.js");
const types_js_1 = require("./types.js");
const constants_js_1 = require("./constants.js");
const node_path_1 = require("node:path");
/**
* Validator regular expressions for various sitemap fields
*/
const allowDeny = /^(?:allow|deny)$/;
exports.validators = {
'price:currency': /^[A-Z]{3}$/,
'price:type': /^(?:rent|purchase|RENT|PURCHASE)$/,
'price:resolution': /^(?:HD|hd|sd|SD)$/,
'platform:relationship': allowDeny,
'restriction:relationship': allowDeny,
restriction: /^([A-Z]{2}( +[A-Z]{2})*)?$/,
platform: /^((web|mobile|tv)( (web|mobile|tv))*)?$/,
// Language codes: zh-cn, zh-tw, or ISO 639 2-3 letter codes
language: /^(zh-cn|zh-tw|[a-z]{2,3})$/,
genres: /^(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated)(, *(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated))*$/,
stock_tickers: /^(\w+:\w+(, *\w+:\w+){0,4})?$/,
};
/**
* Type guard to check if a string is a valid price type
*/
function isPriceType(pt) {
return exports.validators['price:type'].test(pt);
}
/**
* Type guard to check if a string is a valid resolution
*/
function isResolution(res) {
return exports.validators['price:resolution'].test(res);
}
/**
* Type guard to check if a string is a valid changefreq value
*/
const CHANGEFREQ = Object.values(types_js_1.EnumChangefreq);
function isValidChangeFreq(freq) {
return CHANGEFREQ.includes(freq);
}
/**
* Type guard to check if a string is a valid yes/no value
*/
function isValidYesNo(yn) {
return /^YES|NO|[Yy]es|[Nn]o$/.test(yn);
}
/**
* Type guard to check if a string is a valid allow/deny value
*/
function isAllowDeny(ad) {
return allowDeny.test(ad);
}
/**
* Validates that a URL is well-formed and meets security requirements
*
* Security: This function enforces that URLs use safe protocols (http/https),
* are within reasonable length limits (2048 chars per sitemaps.org spec),
* and can be properly parsed. This prevents protocol injection attacks and
* ensures compliance with sitemap specifications.
*
* @param url - The URL to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidHostnameError} If the URL is invalid
*/
function validateURL(url, paramName) {
if (!url || typeof url !== 'string') {
throw new errors_js_1.InvalidHostnameError(url, `${paramName} must be a non-empty string`);
}
if (url.length > constants_js_1.LIMITS.MAX_URL_LENGTH) {
throw new errors_js_1.InvalidHostnameError(url, `${paramName} exceeds maximum length of ${constants_js_1.LIMITS.MAX_URL_LENGTH} characters`);
}
if (!constants_js_1.LIMITS.URL_PROTOCOL_REGEX.test(url)) {
throw new errors_js_1.InvalidHostnameError(url, `${paramName} must use http:// or https:// protocol`);
}
// Validate URL can be parsed
try {
new URL(url);
}
catch (err) {
throw new errors_js_1.InvalidHostnameError(url, `${paramName} is not a valid URL: ${err instanceof Error ? err.message : String(err)}`);
}
}
/**
* Validates that a path doesn't contain path traversal sequences
*
* Security: This function prevents path traversal attacks by detecting
* any occurrence of '..' in the path, whether it appears as '../', '/..',
* or standalone. This prevents attackers from accessing files outside
* the intended directory structure.
*
* @param path - The path to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidPathError} If the path contains traversal sequences
*/
function validatePath(path, paramName) {
if (!path || typeof path !== 'string') {
throw new errors_js_1.InvalidPathError(path, `${paramName} must be a non-empty string`);
}
// Reject absolute paths to prevent arbitrary write location when caller input
// reaches destinationDir (BB-04)
if ((0, node_path_1.isAbsolute)(path)) {
throw new errors_js_1.InvalidPathError(path, `${paramName} must be a relative path (absolute paths are not allowed)`);
}
// Check for path traversal sequences - must check before and after normalization
// to catch both Windows-style (\) and Unix-style (/) separators
if (path.includes('..')) {
throw new errors_js_1.InvalidPathError(path, `${paramName} contains path traversal sequence (..)`);
}
// Additional check after normalization to catch encoded or obfuscated attempts
const normalizedPath = path.replace(/\\/g, '/');
const pathComponents = normalizedPath.split('/').filter((p) => p.length > 0);
if (pathComponents.includes('..')) {
throw new errors_js_1.InvalidPathError(path, `${paramName} contains path traversal sequence (..)`);
}
// Check for null bytes (security issue in some contexts)
if (path.includes('\0')) {
throw new errors_js_1.InvalidPathError(path, `${paramName} contains null byte character`);
}
}
/**
* Validates that a public base path is safe for URL construction
*
* Security: This function prevents path traversal attacks and validates
* that the path is safe for use in URL construction within sitemap indexes.
* It checks for '..' sequences, null bytes, and invalid whitespace that
* could be used to manipulate URL structure or inject malicious content.
*
* @param publicBasePath - The public base path to validate
* @throws {InvalidPublicBasePathError} If the path is invalid
*/
function validatePublicBasePath(publicBasePath) {
if (!publicBasePath || typeof publicBasePath !== 'string') {
throw new errors_js_1.InvalidPublicBasePathError(publicBasePath, 'must be a non-empty string');
}
// Check for path traversal - check the raw string first
if (publicBasePath.includes('..')) {
throw new errors_js_1.InvalidPublicBasePathError(publicBasePath, 'contains path traversal sequence (..)');
}
// Additional check for path components after normalization
const normalizedPath = publicBasePath.replace(/\\/g, '/');
const pathComponents = normalizedPath.split('/').filter((p) => p.length > 0);
if (pathComponents.includes('..')) {
throw new errors_js_1.InvalidPublicBasePathError(publicBasePath, 'contains path traversal sequence (..)');
}
// Check for null bytes
if (publicBasePath.includes('\0')) {
throw new errors_js_1.InvalidPublicBasePathError(publicBasePath, 'contains null byte character');
}
// Check for potentially dangerous characters that could break URL construction
if (/[\r\n\t]/.test(publicBasePath)) {
throw new errors_js_1.InvalidPublicBasePathError(publicBasePath, 'contains invalid whitespace characters');
}
}
/**
* Validates that a limit is within acceptable range per sitemaps.org spec
*
* Security: This function enforces sitemap size limits (1-50,000 URLs per
* sitemap) as specified by sitemaps.org. This prevents resource exhaustion
* attacks and ensures compliance with search engine requirements.
*
* @param limit - The limit to validate
* @throws {InvalidLimitError} If the limit is out of range
*/
function validateLimit(limit) {
if (typeof limit !== 'number' ||
!Number.isFinite(limit) ||
Number.isNaN(limit)) {
throw new errors_js_1.InvalidLimitError(limit);
}
if (limit < constants_js_1.LIMITS.MIN_SITEMAP_ITEM_LIMIT ||
limit > constants_js_1.LIMITS.MAX_SITEMAP_ITEM_LIMIT) {
throw new errors_js_1.InvalidLimitError(limit);
}
// Ensure it's an integer
if (!Number.isInteger(limit)) {
throw new errors_js_1.InvalidLimitError(limit);
}
}
/**
* Validates that an XSL URL is safe and well-formed
*
* Security: This function validates XSL stylesheet URLs to prevent
* injection attacks. It blocks dangerous protocols and content patterns
* that could be used for XSS or other attacks. The validation uses
* case-insensitive matching to catch obfuscated attacks.
*
* @param xslUrl - The XSL URL to validate
* @throws {InvalidXSLUrlError} If the URL is invalid
*/
function validateXSLUrl(xslUrl) {
if (!xslUrl || typeof xslUrl !== 'string') {
throw new errors_js_1.InvalidXSLUrlError(xslUrl, 'must be a non-empty string');
}
if (xslUrl.length > constants_js_1.LIMITS.MAX_URL_LENGTH) {
throw new errors_js_1.InvalidXSLUrlError(xslUrl, `exceeds maximum length of ${constants_js_1.LIMITS.MAX_URL_LENGTH} characters`);
}
if (!constants_js_1.LIMITS.URL_PROTOCOL_REGEX.test(xslUrl)) {
throw new errors_js_1.InvalidXSLUrlError(xslUrl, 'must use http:// or https:// protocol');
}
// Validate URL can be parsed
try {
new URL(xslUrl);
}
catch (err) {
throw new errors_js_1.InvalidXSLUrlError(xslUrl, `is not a valid URL: ${err instanceof Error ? err.message : String(err)}`);
}
// Check for potentially dangerous content (case-insensitive)
const lowerUrl = xslUrl.toLowerCase();
// Block dangerous HTML/script content
if (lowerUrl.includes('<script')) {
throw new errors_js_1.InvalidXSLUrlError(xslUrl, 'contains potentially malicious content (<script tag)');
}
// Block dangerous protocols (already checked http/https above, but double-check for encoded variants)
const dangerousProtocols = [
'javascript:',
'data:',
'vbscript:',
'file:',
'about:',
];
for (const protocol of dangerousProtocols) {
if (lowerUrl.includes(protocol)) {
throw new errors_js_1.InvalidXSLUrlError(xslUrl, `contains dangerous protocol: ${protocol}`);
}
}
// Check for URL-encoded variants of dangerous patterns
// %3C = '<', %3E = '>', %3A = ':'
const encodedPatterns = [
'%3cscript', // <script
'%3c%73%63%72%69%70%74', // <script (fully encoded)
'javascript%3a', // javascript:
'data%3a', // data:
];
for (const pattern of encodedPatterns) {
if (lowerUrl.includes(pattern)) {
throw new errors_js_1.InvalidXSLUrlError(xslUrl, 'contains URL-encoded malicious content');
}
}
// Reject unencoded XML special characters — these must be percent-encoded in
// valid URLs and could break out of XML attribute context if left raw.
if (xslUrl.includes('"') || xslUrl.includes('<') || xslUrl.includes('>')) {
throw new errors_js_1.InvalidXSLUrlError(xslUrl, 'contains unencoded XML special characters (" < >); percent-encode them in the URL');
}
}
/**
* Internal helper to validate fields against their validators
*/
function validate(subject, name, url, level) {
Object.keys(subject).forEach((key) => {
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
const val = subject[key];
if (exports.validators[key] && !exports.validators[key].test(val)) {
if (level === types_js_1.ErrorLevel.THROW) {
throw new errors_js_1.InvalidAttrValue(key, val, exports.validators[key]);
}
else {
console.warn(`${url}: ${name} key ${key} has invalid value: ${val}`);
}
}
});
}
/**
* Internal helper to handle errors based on error level
*/
function handleError(error, level) {
if (level === types_js_1.ErrorLevel.THROW) {
throw error;
}
else if (level === types_js_1.ErrorLevel.WARN) {
console.warn(error.name, error.message);
}
}
/**
* Verifies all data passed in will comply with sitemap spec.
* @param conf Options to validate
* @param level logging level
* @param errorHandler error handling func
*/
function validateSMIOptions(conf, level = types_js_1.ErrorLevel.WARN, errorHandler = handleError) {
if (!conf) {
throw new errors_js_1.NoConfigError();
}
if (level === types_js_1.ErrorLevel.SILENT) {
return conf;
}
const { url, changefreq, priority, news, video } = conf;
if (!url) {
errorHandler(new errors_js_1.NoURLError(), level);
}
if (changefreq) {
if (!isValidChangeFreq(changefreq)) {
errorHandler(new errors_js_1.ChangeFreqInvalidError(url, changefreq), level);
}
}
if (priority) {
if (!(priority >= 0.0 && priority <= 1.0)) {
errorHandler(new errors_js_1.PriorityInvalidError(url, priority), level);
}
}
if (news) {
if (news.access &&
news.access !== 'Registration' &&
news.access !== 'Subscription') {
errorHandler(new errors_js_1.InvalidNewsAccessValue(url, news.access), level);
}
if (!news.publication ||
!news.publication.name ||
!news.publication.language ||
!news.publication_date ||
!news.title) {
errorHandler(new errors_js_1.InvalidNewsFormat(url), level);
}
validate(news, 'news', url, level);
validate(news.publication, 'publication', url, level);
}
if (video) {
video.forEach((vid) => {
if (vid.duration !== undefined) {
if (vid.duration < 0 || vid.duration > 28800) {
errorHandler(new errors_js_1.InvalidVideoDuration(url, vid.duration), level);
}
}
if (vid.rating !== undefined && (vid.rating < 0 || vid.rating > 5)) {
errorHandler(new errors_js_1.InvalidVideoRating(url, vid.title, vid.rating), level);
}
if (typeof vid !== 'object' ||
!vid.thumbnail_loc ||
!vid.title ||
!vid.description) {
// has to be an object and include required categories https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190
errorHandler(new errors_js_1.InvalidVideoFormat(url), level);
}
if (vid.title.length > 100) {
errorHandler(new errors_js_1.InvalidVideoTitle(url, vid.title.length), level);
}
if (vid.description.length > 2048) {
errorHandler(new errors_js_1.InvalidVideoDescription(url, vid.description.length), level);
}
if (vid.view_count !== undefined && vid.view_count < 0) {
errorHandler(new errors_js_1.InvalidVideoViewCount(url, vid.view_count), level);
}
if (vid.tag.length > 32) {
errorHandler(new errors_js_1.InvalidVideoTagCount(url, vid.tag.length), level);
}
if (vid.category !== undefined && vid.category?.length > 256) {
errorHandler(new errors_js_1.InvalidVideoCategory(url, vid.category.length), level);
}
if (vid.family_friendly !== undefined &&
!isValidYesNo(vid.family_friendly)) {
errorHandler(new errors_js_1.InvalidVideoFamilyFriendly(url, vid.family_friendly), level);
}
if (vid.restriction) {
if (!exports.validators.restriction.test(vid.restriction)) {
errorHandler(new errors_js_1.InvalidVideoRestriction(url, vid.restriction), level);
}
if (!vid['restriction:relationship'] ||
!isAllowDeny(vid['restriction:relationship'])) {
errorHandler(new errors_js_1.InvalidVideoRestrictionRelationship(url, vid['restriction:relationship']), level);
}
}
// TODO price element should be unbounded
if ((vid.price === '' && vid['price:type'] === undefined) ||
(vid['price:type'] !== undefined && !isPriceType(vid['price:type']))) {
errorHandler(new errors_js_1.InvalidVideoPriceType(url, vid['price:type'], vid.price), level);
}
if (vid['price:resolution'] !== undefined &&
!isResolution(vid['price:resolution'])) {
errorHandler(new errors_js_1.InvalidVideoResolution(url, vid['price:resolution']), level);
}
if (vid['price:currency'] !== undefined &&
!exports.validators['price:currency'].test(vid['price:currency'])) {
errorHandler(new errors_js_1.InvalidVideoPriceCurrency(url, vid['price:currency']), level);
}
validate(vid, 'video', url, level);
});
}
return conf;
}
node_modules/sitemap/dist/cjs/lib/xmllint.d.ts
Generated Vendored
+12
View File
@@ -0,0 +1,12 @@
import { Readable } from 'node:stream';
/**
* Verify the passed in xml is valid. Requires xmllib be installed
*
* Security: This function always pipes XML content via stdin to prevent
* command injection vulnerabilities. Never pass user-controlled strings
* as file path arguments to xmllint.
*
* @param xml what you want validated (string or Readable stream)
* @return {Promise<void>} resolves on valid rejects [error stderr]
*/
export declare function xmlLint(xml: string | Readable): Promise<void>;
node_modules/sitemap/dist/cjs/lib/xmllint.js
Generated Vendored
+81
View File
@@ -0,0 +1,81 @@
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.xmlLint = xmlLint;
const node_fs_1 = require("node:fs");
const node_path_1 = require("node:path");
const node_child_process_1 = require("node:child_process");
const errors_js_1 = require("./errors.js");
/**
* Finds the `schema` directory with robust path resolution.
* Searches from the project root directory using process.cwd().
* This works correctly regardless of whether the code is running from:
* - Source: lib/xmllint.ts
* - ESM build: dist/esm/lib/xmllint.js
* - CJS build: dist/cjs/lib/xmllint.js
* - Test environment
*
* @throws {Error} if the schema directory is not found
* @returns {string} the path to the schema directory
*/
function findSchemaDir() {
// Search for schema directory from project root
// This works in test, build, and source environments
const possiblePaths = [
(0, node_path_1.resolve)(process.cwd(), 'schema'), // From project root
(0, node_path_1.resolve)(process.cwd(), '..', 'schema'), // One level up
(0, node_path_1.resolve)(process.cwd(), '..', '..', 'schema'), // Two levels up
];
for (const schemaPath of possiblePaths) {
if ((0, node_fs_1.existsSync)(schemaPath)) {
return schemaPath;
}
}
throw new Error(`Schema directory not found. Searched paths: ${possiblePaths.join(', ')}`);
}
/**
* Verify the passed in xml is valid. Requires xmllib be installed
*
* Security: This function always pipes XML content via stdin to prevent
* command injection vulnerabilities. Never pass user-controlled strings
* as file path arguments to xmllint.
*
* @param xml what you want validated (string or Readable stream)
* @return {Promise<void>} resolves on valid rejects [error stderr]
*/
function xmlLint(xml) {
const args = [
'--schema',
(0, node_path_1.resolve)(findSchemaDir(), 'all.xsd'),
'--noout',
'-', // Always read from stdin for security
];
return new Promise((resolve, reject) => {
(0, node_child_process_1.execFile)('which', ['xmllint'], (error, stdout, stderr) => {
if (error) {
reject([new errors_js_1.XMLLintUnavailable()]);
return;
}
const xmllint = (0, node_child_process_1.execFile)('xmllint', args, (error, stdout, stderr) => {
if (error) {
reject([error, stderr]);
}
resolve();
});
// Always pipe XML content via stdin for security
if (xmllint.stdin) {
if (typeof xml === 'string') {
// Convert string to stream and pipe to stdin
xmllint.stdin.write(xml);
xmllint.stdin.end();
}
else if (xml) {
// Pipe readable stream to stdin
xml.pipe(xmllint.stdin);
}
}
if (xmllint.stdout) {
xmllint.stdout.unpipe();
}
});
});
}
node_modules/sitemap/dist/cjs/package.json
Generated Vendored
+1
View File
@@ -0,0 +1 @@
{"type":"commonjs"}
node_modules/sitemap/dist/esm/cli.d.ts
Generated Vendored
+2
View File
@@ -0,0 +1,2 @@
#!/usr/bin/env node
export {};
node_modules/sitemap/dist/esm/cli.js
Generated Vendored Executable
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env node
import { createReadStream, createWriteStream } from 'node:fs';
import { readFileSync } from 'node:fs';
import { resolve } from 'node:path';
import { xmlLint } from './lib/xmllint.js';
import { XMLLintUnavailable } from './lib/errors.js';
import { ObjectStreamToJSON, XMLToSitemapItemStream, } from './lib/sitemap-parser.js';
import { lineSeparatedURLsToSitemapOptions } from './lib/utils.js';
import { SitemapStream } from './lib/sitemap-stream.js';
import { SitemapAndIndexStream } from './lib/sitemap-index-stream.js';
import { URL } from 'node:url';
import { createGzip } from 'node:zlib';
import { ErrorLevel } from './lib/types.js';
import arg from 'arg';
// Read package.json from the project root (one level up from dist/esm or dist/cjs)
// In ESM, __dirname is not defined, so we use import.meta.url
// In CJS, __dirname is defined and import.meta is not available
let currentDir;
try {
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore - __dirname may not be defined in ESM
currentDir = __dirname;
}
catch {
// ESM fallback using import.meta.url
currentDir = new URL('.', import.meta.url).pathname;
}
const packageJson = JSON.parse(readFileSync(resolve(currentDir, '../../package.json'), 'utf8'));
const pickStreamOrArg = (argv) => {
if (!argv._.length) {
return process.stdin;
}
else {
return createReadStream(argv._[0], { encoding: 'utf8' });
}
};
const argSpec = {
'--help': Boolean,
'--version': Boolean,
'--validate': Boolean,
'--index': Boolean,
'--index-base-url': String,
'--limit': Number,
'--parse': Boolean,
'--single-line-json': Boolean,
'--prepend': String,
'--gzip': Boolean,
'-h': '--help',
};
const argv = arg(argSpec);
function getStream() {
if (argv._ && argv._.length) {
return createReadStream(argv._[0]);
}
else {
console.warn('Reading from stdin. If you are not piping anything in, this command is not doing anything');
return process.stdin;
}
}
if (argv['--version']) {
console.log(packageJson.version);
}
else if (argv['--help']) {
console.log(`
Turn a list of urls into a sitemap xml.
Options:
--help Print this text
--version Print the version
--validate Ensure the passed in file is conforms to the sitemap spec
--index Create an index and stream that out. Writes out sitemaps along the way.
--index-base-url Base url the sitemaps will be hosted eg. https://example.com/sitemaps/
--limit=45000 Set a custom limit to the items per sitemap
--parse Parse fed xml and spit out config
--prepend=sitemap.xml Prepend the streamed in sitemap configs to sitemap.xml
--gzip Compress output
--single-line-json When used with parse, it spits out each entry as json rather than the whole json.
# examples
Generate a sitemap index file as well as sitemaps
npx sitemap --gzip --index --index-base-url https://example.com/path/to/sitemaps/ < listofurls.txt > sitemap-index.xml.gz
Add to a sitemap
npx sitemap --prepend sitemap.xml < listofurls.json
Turn an existing sitemap into configuration understood by the sitemap library
npx sitemap --parse sitemap.xml
Use XMLLib to validate your sitemap (requires xmllib)
npx sitemap --validate sitemap.xml
`);
}
else if (argv['--parse']) {
let oStream = getStream()
.pipe(new XMLToSitemapItemStream({ level: ErrorLevel.THROW }))
.pipe(new ObjectStreamToJSON({ lineSeparated: !argv['--single-line-json'] }));
if (argv['--gzip']) {
oStream = oStream.pipe(createGzip());
}
oStream.pipe(process.stdout);
}
else if (argv['--validate']) {
xmlLint(getStream())
.then(() => console.log('valid'))
.catch(([error, stderr]) => {
if (error instanceof XMLLintUnavailable) {
console.error(error.message);
return;
}
else {
console.log(stderr);
}
});
}
else if (argv['--index']) {
const limit = argv['--limit'];
const baseURL = argv['--index-base-url'];
if (!baseURL) {
throw new Error("You must specify where the sitemaps will be hosted. use --index-base-url 'https://example.com/path'");
}
const sms = new SitemapAndIndexStream({
limit,
getSitemapStream: (i) => {
const sm = new SitemapStream();
const path = `./sitemap-${i}.xml`;
let ws;
if (argv['--gzip']) {
ws = sm.pipe(createGzip()).pipe(createWriteStream(path));
}
else {
ws = sm.pipe(createWriteStream(path));
}
return [new URL(path, baseURL).toString(), sm, ws];
},
});
let oStream = lineSeparatedURLsToSitemapOptions(pickStreamOrArg(argv)).pipe(sms);
if (argv['--gzip']) {
oStream = oStream.pipe(createGzip());
}
oStream.pipe(process.stdout);
}
else {
const sms = new SitemapStream();
if (argv['--prepend']) {
createReadStream(argv['--prepend'])
.pipe(new XMLToSitemapItemStream())
.pipe(sms);
}
const oStream = lineSeparatedURLsToSitemapOptions(pickStreamOrArg(argv)).pipe(sms);
if (argv['--gzip']) {
oStream.pipe(createGzip()).pipe(process.stdout);
}
else {
oStream.pipe(process.stdout);
}
}
node_modules/sitemap/dist/esm/index.d.ts
Generated Vendored
+17
View File
@@ -0,0 +1,17 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
export { SitemapItemStream, SitemapItemStreamOptions, } from './lib/sitemap-item-stream.js';
export { IndexTagNames, SitemapIndexStream, SitemapIndexStreamOptions, SitemapAndIndexStream, SitemapAndIndexStreamOptions, } from './lib/sitemap-index-stream.js';
export { streamToPromise, SitemapStream, SitemapStreamOptions, } from './lib/sitemap-stream.js';
export * from './lib/errors.js';
export * from './lib/types.js';
export { lineSeparatedURLsToSitemapOptions, mergeStreams, validateSMIOptions, normalizeURL, ReadlineStream, ReadlineStreamOptions, } from './lib/utils.js';
export { xmlLint } from './lib/xmllint.js';
export { parseSitemap, XMLToSitemapItemStream, XMLToSitemapItemStreamOptions, ObjectStreamToJSON, ObjectStreamToJSONOptions, } from './lib/sitemap-parser.js';
export { parseSitemapIndex, XMLToSitemapIndexStream, XMLToSitemapIndexItemStreamOptions, IndexObjectStreamToJSON, IndexObjectStreamToJSONOptions, } from './lib/sitemap-index-parser.js';
export { simpleSitemapAndIndex, SimpleSitemapAndIndexOptions, } from './lib/sitemap-simple.js';
export { validateURL, validatePath, validateLimit, validatePublicBasePath, validateXSLUrl, validators, isPriceType, isResolution, isValidChangeFreq, isValidYesNo, isAllowDeny, } from './lib/validation.js';
export { LIMITS, DEFAULT_SITEMAP_ITEM_LIMIT } from './lib/constants.js';
node_modules/sitemap/dist/esm/index.js
Generated Vendored
+17
View File
@@ -0,0 +1,17 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
export { SitemapItemStream, } from './lib/sitemap-item-stream.js';
export { IndexTagNames, SitemapIndexStream, SitemapAndIndexStream, } from './lib/sitemap-index-stream.js';
export { streamToPromise, SitemapStream, } from './lib/sitemap-stream.js';
export * from './lib/errors.js';
export * from './lib/types.js';
export { lineSeparatedURLsToSitemapOptions, mergeStreams, validateSMIOptions, normalizeURL, ReadlineStream, } from './lib/utils.js';
export { xmlLint } from './lib/xmllint.js';
export { parseSitemap, XMLToSitemapItemStream, ObjectStreamToJSON, } from './lib/sitemap-parser.js';
export { parseSitemapIndex, XMLToSitemapIndexStream, IndexObjectStreamToJSON, } from './lib/sitemap-index-parser.js';
export { simpleSitemapAndIndex, } from './lib/sitemap-simple.js';
export { validateURL, validatePath, validateLimit, validatePublicBasePath, validateXSLUrl, validators, isPriceType, isResolution, isValidChangeFreq, isValidYesNo, isAllowDeny, } from './lib/validation.js';
export { LIMITS, DEFAULT_SITEMAP_ITEM_LIMIT } from './lib/constants.js';
node_modules/sitemap/dist/esm/lib/constants.d.ts
Generated Vendored
+49
View File
@@ -0,0 +1,49 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
/**
* Shared constants used across the sitemap library
* This file serves as a single source of truth for limits and validation patterns
*/
/**
* Security limits for sitemap generation and parsing
*
* These limits are based on:
* - sitemaps.org protocol specification
* - Security best practices to prevent DoS and injection attacks
* - Google's sitemap extension specifications
*
* @see https://www.sitemaps.org/protocol.html
* @see https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap
*/
export declare const LIMITS: {
readonly MAX_URL_LENGTH: 2048;
readonly URL_PROTOCOL_REGEX: RegExp;
readonly MIN_SITEMAP_ITEM_LIMIT: 1;
readonly MAX_SITEMAP_ITEM_LIMIT: 50000;
readonly MAX_VIDEO_TITLE_LENGTH: 100;
readonly MAX_VIDEO_DESCRIPTION_LENGTH: 2048;
readonly MAX_VIDEO_CATEGORY_LENGTH: 256;
readonly MAX_TAGS_PER_VIDEO: 32;
readonly MAX_NEWS_TITLE_LENGTH: 200;
readonly MAX_NEWS_NAME_LENGTH: 256;
readonly MAX_IMAGE_CAPTION_LENGTH: 512;
readonly MAX_IMAGE_TITLE_LENGTH: 512;
readonly MAX_IMAGES_PER_URL: 1000;
readonly MAX_VIDEOS_PER_URL: 100;
readonly MAX_LINKS_PER_URL: 100;
readonly MAX_URL_ENTRIES: 50000;
readonly ISO_DATE_REGEX: RegExp;
readonly MAX_CUSTOM_NAMESPACES: 20;
readonly MAX_NAMESPACE_LENGTH: 512;
readonly MAX_PARSER_ERRORS: 100;
};
/**
* Default maximum number of items in each sitemap XML file
* Set below the max to leave room for URLs added during processing
*
* @see https://www.sitemaps.org/protocol.html#index
*/
export declare const DEFAULT_SITEMAP_ITEM_LIMIT = 45000;
node_modules/sitemap/dist/esm/lib/constants.js
Generated Vendored
+60
View File
@@ -0,0 +1,60 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
/**
* Shared constants used across the sitemap library
* This file serves as a single source of truth for limits and validation patterns
*/
/**
* Security limits for sitemap generation and parsing
*
* These limits are based on:
* - sitemaps.org protocol specification
* - Security best practices to prevent DoS and injection attacks
* - Google's sitemap extension specifications
*
* @see https://www.sitemaps.org/protocol.html
* @see https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap
*/
export const LIMITS = {
// URL constraints per sitemaps.org spec
MAX_URL_LENGTH: 2048,
URL_PROTOCOL_REGEX: /^https?:\/\//i,
// Sitemap size limits per sitemaps.org spec
MIN_SITEMAP_ITEM_LIMIT: 1,
MAX_SITEMAP_ITEM_LIMIT: 50000,
// Video field length constraints per Google spec
MAX_VIDEO_TITLE_LENGTH: 100,
MAX_VIDEO_DESCRIPTION_LENGTH: 2048,
MAX_VIDEO_CATEGORY_LENGTH: 256,
MAX_TAGS_PER_VIDEO: 32,
// News field length constraints per Google spec
MAX_NEWS_TITLE_LENGTH: 200,
MAX_NEWS_NAME_LENGTH: 256,
// Image field length constraints per Google spec
MAX_IMAGE_CAPTION_LENGTH: 512,
MAX_IMAGE_TITLE_LENGTH: 512,
// Limits on number of items per URL entry
MAX_IMAGES_PER_URL: 1000,
MAX_VIDEOS_PER_URL: 100,
MAX_LINKS_PER_URL: 100,
// Total entries in a sitemap
MAX_URL_ENTRIES: 50000,
// Date validation - ISO 8601 / W3C format
ISO_DATE_REGEX: /^\d{4}-\d{2}-\d{2}(T\d{2}:\d{2}:\d{2}(\.\d{3})?([+-]\d{2}:\d{2}|Z)?)?$/,
// Custom namespace limits to prevent DoS
MAX_CUSTOM_NAMESPACES: 20,
MAX_NAMESPACE_LENGTH: 512,
// Cap on stored parser errors to prevent memory DoS (BB-03)
// Errors beyond this limit are counted in errorCount but not retained as objects
MAX_PARSER_ERRORS: 100,
};
/**
* Default maximum number of items in each sitemap XML file
* Set below the max to leave room for URLs added during processing
*
* @see https://www.sitemaps.org/protocol.html#index
*/
export const DEFAULT_SITEMAP_ITEM_LIMIT = 45000;
node_modules/sitemap/dist/esm/lib/errors.d.ts
Generated Vendored
+116
View File
@@ -0,0 +1,116 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
/**
* URL in SitemapItem does not exist
*/
export declare class NoURLError extends Error {
constructor(message?: string);
}
/**
* Config was not passed to SitemapItem constructor
*/
export declare class NoConfigError extends Error {
constructor(message?: string);
}
/**
* changefreq property in sitemap is invalid
*/
export declare class ChangeFreqInvalidError extends Error {
constructor(url: string, changefreq: any);
}
/**
* priority property in sitemap is invalid
*/
export declare class PriorityInvalidError extends Error {
constructor(url: string, priority: any);
}
/**
* SitemapIndex target Folder does not exists
*/
export declare class UndefinedTargetFolder extends Error {
constructor(message?: string);
}
export declare class InvalidVideoFormat extends Error {
constructor(url: string);
}
export declare class InvalidVideoDuration extends Error {
constructor(url: string, duration: any);
}
export declare class InvalidVideoDescription extends Error {
constructor(url: string, length: number);
}
export declare class InvalidVideoRating extends Error {
constructor(url: string, title: any, rating: any);
}
export declare class InvalidAttrValue extends Error {
constructor(key: string, val: any, validator: RegExp);
}
export declare class InvalidAttr extends Error {
constructor(key: string);
}
export declare class InvalidNewsFormat extends Error {
constructor(url: string);
}
export declare class InvalidNewsAccessValue extends Error {
constructor(url: string, access: any);
}
export declare class XMLLintUnavailable extends Error {
constructor(message?: string);
}
export declare class InvalidVideoTitle extends Error {
constructor(url: string, length: number);
}
export declare class InvalidVideoViewCount extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoTagCount extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoCategory extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoFamilyFriendly extends Error {
constructor(url: string, fam: string);
}
export declare class InvalidVideoRestriction extends Error {
constructor(url: string, code: string);
}
export declare class InvalidVideoRestrictionRelationship extends Error {
constructor(url: string, val?: string);
}
export declare class InvalidVideoPriceType extends Error {
constructor(url: string, priceType?: string, price?: string);
}
export declare class InvalidVideoResolution extends Error {
constructor(url: string, resolution: string);
}
export declare class InvalidVideoPriceCurrency extends Error {
constructor(url: string, currency: string);
}
export declare class EmptyStream extends Error {
constructor();
}
export declare class EmptySitemap extends Error {
constructor();
}
export declare class InvalidPathError extends Error {
constructor(path: string, reason: string);
}
export declare class InvalidHostnameError extends Error {
constructor(hostname: string, reason: string);
}
export declare class InvalidLimitError extends Error {
constructor(limit: any);
}
export declare class InvalidPublicBasePathError extends Error {
constructor(publicBasePath: string, reason: string);
}
export declare class InvalidXSLUrlError extends Error {
constructor(xslUrl: string, reason: string);
}
export declare class InvalidXMLAttributeNameError extends Error {
constructor(attributeName: string);
}
node_modules/sitemap/dist/esm/lib/errors.js
Generated Vendored
+256
View File
@@ -0,0 +1,256 @@
/* eslint-disable @typescript-eslint/no-explicit-any */
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
/**
* URL in SitemapItem does not exist
*/
export class NoURLError extends Error {
constructor(message) {
super(message || 'URL is required');
this.name = 'NoURLError';
Error.captureStackTrace(this, NoURLError);
}
}
/**
* Config was not passed to SitemapItem constructor
*/
export class NoConfigError extends Error {
constructor(message) {
super(message || 'SitemapItem requires a configuration');
this.name = 'NoConfigError';
Error.captureStackTrace(this, NoConfigError);
}
}
/**
* changefreq property in sitemap is invalid
*/
export class ChangeFreqInvalidError extends Error {
constructor(url, changefreq) {
super(`${url}: changefreq "${changefreq}" is invalid`);
this.name = 'ChangeFreqInvalidError';
Error.captureStackTrace(this, ChangeFreqInvalidError);
}
}
/**
* priority property in sitemap is invalid
*/
export class PriorityInvalidError extends Error {
constructor(url, priority) {
super(`${url}: priority "${priority}" must be a number between 0 and 1 inclusive`);
this.name = 'PriorityInvalidError';
Error.captureStackTrace(this, PriorityInvalidError);
}
}
/**
* SitemapIndex target Folder does not exists
*/
export class UndefinedTargetFolder extends Error {
constructor(message) {
super(message || 'Target folder must exist');
this.name = 'UndefinedTargetFolder';
Error.captureStackTrace(this, UndefinedTargetFolder);
}
}
export class InvalidVideoFormat extends Error {
constructor(url) {
super(`${url} video must include thumbnail_loc, title and description fields for videos`);
this.name = 'InvalidVideoFormat';
Error.captureStackTrace(this, InvalidVideoFormat);
}
}
export class InvalidVideoDuration extends Error {
constructor(url, duration) {
super(`${url} duration "${duration}" must be an integer of seconds between 0 and 28800`);
this.name = 'InvalidVideoDuration';
Error.captureStackTrace(this, InvalidVideoDuration);
}
}
export class InvalidVideoDescription extends Error {
constructor(url, length) {
const message = `${url}: video description is too long ${length} vs limit of 2048 characters.`;
super(message);
this.name = 'InvalidVideoDescription';
Error.captureStackTrace(this, InvalidVideoDescription);
}
}
export class InvalidVideoRating extends Error {
constructor(url, title, rating) {
super(`${url}: video "${title}" rating "${rating}" must be between 0 and 5 inclusive`);
this.name = 'InvalidVideoRating';
Error.captureStackTrace(this, InvalidVideoRating);
}
}
export class InvalidAttrValue extends Error {
constructor(key, val, validator) {
super('"' +
val +
'" tested against: ' +
validator +
' is not a valid value for attr: "' +
key +
'"');
this.name = 'InvalidAttrValue';
Error.captureStackTrace(this, InvalidAttrValue);
}
}
// InvalidAttr is only thrown when attrbuilder is called incorrectly internally
/* istanbul ignore next */
export class InvalidAttr extends Error {
constructor(key) {
super('"' + key + '" is malformed');
this.name = 'InvalidAttr';
Error.captureStackTrace(this, InvalidAttr);
}
}
export class InvalidNewsFormat extends Error {
constructor(url) {
super(`${url} News must include publication, publication name, publication language, title, and publication_date for news`);
this.name = 'InvalidNewsFormat';
Error.captureStackTrace(this, InvalidNewsFormat);
}
}
export class InvalidNewsAccessValue extends Error {
constructor(url, access) {
super(`${url} News access "${access}" must be either Registration, Subscription or not be present`);
this.name = 'InvalidNewsAccessValue';
Error.captureStackTrace(this, InvalidNewsAccessValue);
}
}
export class XMLLintUnavailable extends Error {
constructor(message) {
super(message || 'xmlLint is not installed. XMLLint is required to validate');
this.name = 'XMLLintUnavailable';
Error.captureStackTrace(this, XMLLintUnavailable);
}
}
export class InvalidVideoTitle extends Error {
constructor(url, length) {
super(`${url}: video title is too long ${length} vs 100 character limit`);
this.name = 'InvalidVideoTitle';
Error.captureStackTrace(this, InvalidVideoTitle);
}
}
export class InvalidVideoViewCount extends Error {
constructor(url, count) {
super(`${url}: video view count must be positive, view count was ${count}`);
this.name = 'InvalidVideoViewCount';
Error.captureStackTrace(this, InvalidVideoViewCount);
}
}
export class InvalidVideoTagCount extends Error {
constructor(url, count) {
super(`${url}: video can have no more than 32 tags, this has ${count}`);
this.name = 'InvalidVideoTagCount';
Error.captureStackTrace(this, InvalidVideoTagCount);
}
}
export class InvalidVideoCategory extends Error {
constructor(url, count) {
super(`${url}: video category can only be 256 characters but was passed ${count}`);
this.name = 'InvalidVideoCategory';
Error.captureStackTrace(this, InvalidVideoCategory);
}
}
export class InvalidVideoFamilyFriendly extends Error {
constructor(url, fam) {
super(`${url}: video family friendly must be yes or no, was passed "${fam}"`);
this.name = 'InvalidVideoFamilyFriendly';
Error.captureStackTrace(this, InvalidVideoFamilyFriendly);
}
}
export class InvalidVideoRestriction extends Error {
constructor(url, code) {
super(`${url}: video restriction must be one or more two letter country codes. Was passed "${code}"`);
this.name = 'InvalidVideoRestriction';
Error.captureStackTrace(this, InvalidVideoRestriction);
}
}
export class InvalidVideoRestrictionRelationship extends Error {
constructor(url, val) {
super(`${url}: video restriction relationship must be either allow or deny. Was passed "${val}"`);
this.name = 'InvalidVideoRestrictionRelationship';
Error.captureStackTrace(this, InvalidVideoRestrictionRelationship);
}
}
export class InvalidVideoPriceType extends Error {
constructor(url, priceType, price) {
super(priceType === undefined && price === ''
? `${url}: video priceType is required when price is not provided`
: `${url}: video price type "${priceType}" is not "rent" or "purchase"`);
this.name = 'InvalidVideoPriceType';
Error.captureStackTrace(this, InvalidVideoPriceType);
}
}
export class InvalidVideoResolution extends Error {
constructor(url, resolution) {
super(`${url}: video price resolution "${resolution}" is not hd or sd`);
this.name = 'InvalidVideoResolution';
Error.captureStackTrace(this, InvalidVideoResolution);
}
}
export class InvalidVideoPriceCurrency extends Error {
constructor(url, currency) {
super(`${url}: video price currency "${currency}" must be a three capital letter abbrieviation for the country currency`);
this.name = 'InvalidVideoPriceCurrency';
Error.captureStackTrace(this, InvalidVideoPriceCurrency);
}
}
export class EmptyStream extends Error {
constructor() {
super('You have ended the stream before anything was written. streamToPromise MUST be called before ending the stream.');
this.name = 'EmptyStream';
Error.captureStackTrace(this, EmptyStream);
}
}
export class EmptySitemap extends Error {
constructor() {
super('You ended the stream without writing anything.');
this.name = 'EmptySitemap';
Error.captureStackTrace(this, EmptyStream);
}
}
export class InvalidPathError extends Error {
constructor(path, reason) {
super(`Invalid path "${path}": ${reason}`);
this.name = 'InvalidPathError';
Error.captureStackTrace(this, InvalidPathError);
}
}
export class InvalidHostnameError extends Error {
constructor(hostname, reason) {
super(`Invalid hostname "${hostname}": ${reason}`);
this.name = 'InvalidHostnameError';
Error.captureStackTrace(this, InvalidHostnameError);
}
}
export class InvalidLimitError extends Error {
constructor(limit) {
super(`Invalid limit "${limit}": must be a number between 1 and 50000 (per sitemaps.org spec)`);
this.name = 'InvalidLimitError';
Error.captureStackTrace(this, InvalidLimitError);
}
}
export class InvalidPublicBasePathError extends Error {
constructor(publicBasePath, reason) {
super(`Invalid publicBasePath "${publicBasePath}": ${reason}`);
this.name = 'InvalidPublicBasePathError';
Error.captureStackTrace(this, InvalidPublicBasePathError);
}
}
export class InvalidXSLUrlError extends Error {
constructor(xslUrl, reason) {
super(`Invalid xslUrl "${xslUrl}": ${reason}`);
this.name = 'InvalidXSLUrlError';
Error.captureStackTrace(this, InvalidXSLUrlError);
}
}
export class InvalidXMLAttributeNameError extends Error {
constructor(attributeName) {
super(`Invalid XML attribute name "${attributeName}": must contain only alphanumeric characters, hyphens, underscores, and colons`);
this.name = 'InvalidXMLAttributeNameError';
Error.captureStackTrace(this, InvalidXMLAttributeNameError);
}
}
node_modules/sitemap/dist/esm/lib/sitemap-index-parser.d.ts
Generated Vendored
+55
View File
@@ -0,0 +1,55 @@
import type { SAXStream } from 'sax';
import { Readable, Transform, TransformOptions, TransformCallback } from 'node:stream';
import { IndexItem, ErrorLevel } from './types.js';
type Logger = (level: 'warn' | 'error' | 'info' | 'log', ...message: Parameters<Console['log']>) => void;
export interface XMLToSitemapIndexItemStreamOptions extends TransformOptions {
level?: ErrorLevel;
logger?: Logger | false;
}
/**
* Takes a stream of xml and transforms it into a stream of IndexItems
* Use this to parse existing sitemap indices into config options compatible with this library
*/
export declare class XMLToSitemapIndexStream extends Transform {
level: ErrorLevel;
logger: Logger;
error: Error | null;
saxStream: SAXStream;
constructor(opts?: XMLToSitemapIndexItemStreamOptions);
_transform(data: string, encoding: string, callback: TransformCallback): void;
private err;
}
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
const { createReadStream } = require('fs')
const { parseSitemapIndex, createSitemap } = require('sitemap')
parseSitemapIndex(createReadStream('./example-index.xml')).then(
// produces the same xml
// you can, of course, more practically modify it or store it
(xmlConfig) => console.log(createSitemap(xmlConfig).toString()),
(err) => console.log(err)
)
```
@param {Readable} xml what to parse
@param {number} maxEntries Maximum number of sitemap entries to parse (default: 50,000 per sitemaps.org spec)
@return {Promise<IndexItem[]>} resolves with list of index items that can be fed into a SitemapIndexStream. Rejects with an Error object.
*/
export declare function parseSitemapIndex(xml: Readable, maxEntries?: number): Promise<IndexItem[]>;
export interface IndexObjectStreamToJSONOptions extends TransformOptions {
lineSeparated: boolean;
}
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
export declare class IndexObjectStreamToJSON extends Transform {
lineSeparated: boolean;
firstWritten: boolean;
constructor(opts?: IndexObjectStreamToJSONOptions);
_transform(chunk: IndexItem, encoding: string, cb: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
export {};
node_modules/sitemap/dist/esm/lib/sitemap-index-parser.js
Generated Vendored
+262
View File
@@ -0,0 +1,262 @@
import sax from 'sax';
import { Transform, } from 'node:stream';
import { ErrorLevel, IndexTagNames } from './types.js';
import { validateURL } from './validation.js';
import { LIMITS } from './constants.js';
function isValidTagName(tagName) {
// This only works because the enum name and value are the same
return tagName in IndexTagNames;
}
function tagTemplate() {
return {
url: '',
};
}
const defaultLogger = (level, ...message) => console[level](...message);
const defaultStreamOpts = {
logger: defaultLogger,
};
/**
* Takes a stream of xml and transforms it into a stream of IndexItems
* Use this to parse existing sitemap indices into config options compatible with this library
*/
export class XMLToSitemapIndexStream extends Transform {
level;
logger;
error;
saxStream;
constructor(opts = defaultStreamOpts) {
opts.objectMode = true;
super(opts);
this.error = null;
this.saxStream = sax.createStream(true, {
xmlns: true,
// @ts-expect-error - SAX types don't include strictEntities option
strictEntities: true,
trim: true,
});
this.level = opts.level || ErrorLevel.WARN;
if (this.level !== ErrorLevel.SILENT && opts.logger !== false) {
this.logger = opts.logger ?? defaultLogger;
}
else {
this.logger = () => undefined;
}
let currentItem = tagTemplate();
let currentTag;
this.saxStream.on('opentagstart', (tag) => {
currentTag = tag.name;
});
this.saxStream.on('opentag', (tag) => {
if (!isValidTagName(tag.name)) {
this.logger('warn', 'unhandled tag', tag.name);
this.err(`unhandled tag: ${tag.name}`);
}
});
this.saxStream.on('text', (text) => {
switch (currentTag) {
case IndexTagNames.loc:
// Validate URL for security: prevents protocol injection, checks length limits
try {
validateURL(text, 'Sitemap index URL');
currentItem.url = text;
}
catch (error) {
const errMsg = error instanceof Error ? error.message : String(error);
this.logger('warn', 'Invalid URL in sitemap index:', errMsg);
this.err(`Invalid URL in sitemap index: ${errMsg}`);
}
break;
case IndexTagNames.lastmod:
// Validate date format for security and spec compliance
if (text && !LIMITS.ISO_DATE_REGEX.test(text)) {
this.logger('warn', 'Invalid lastmod date format in sitemap index:', text);
this.err(`Invalid lastmod date format: ${text}`);
}
else {
currentItem.lastmod = text;
}
break;
default:
this.logger('log', 'unhandled text for tag:', currentTag, `'${text}'`);
this.err(`unhandled text for tag: ${currentTag} '${text}'`);
break;
}
});
this.saxStream.on('cdata', (text) => {
switch (currentTag) {
case IndexTagNames.loc:
// Validate URL for security: prevents protocol injection, checks length limits
try {
validateURL(text, 'Sitemap index URL');
currentItem.url = text;
}
catch (error) {
const errMsg = error instanceof Error ? error.message : String(error);
this.logger('warn', 'Invalid URL in sitemap index:', errMsg);
this.err(`Invalid URL in sitemap index: ${errMsg}`);
}
break;
case IndexTagNames.lastmod:
// Validate date format for security and spec compliance
if (text && !LIMITS.ISO_DATE_REGEX.test(text)) {
this.logger('warn', 'Invalid lastmod date format in sitemap index:', text);
this.err(`Invalid lastmod date format: ${text}`);
}
else {
currentItem.lastmod = text;
}
break;
default:
this.logger('log', 'unhandled cdata for tag:', currentTag);
this.err(`unhandled cdata for tag: ${currentTag}`);
break;
}
});
this.saxStream.on('attribute', (attr) => {
switch (currentTag) {
case IndexTagNames.sitemapindex:
break;
default:
this.logger('log', 'unhandled attr', currentTag, attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
});
this.saxStream.on('closetag', (tag) => {
switch (tag) {
case IndexTagNames.sitemap:
// Only push items with valid URLs (non-empty after validation)
if (currentItem.url) {
this.push(currentItem);
}
currentItem = tagTemplate();
break;
default:
break;
}
});
}
_transform(data, encoding, callback) {
try {
const cb = () => callback(this.level === ErrorLevel.THROW ? this.error : null);
// correcting the type here can be done without making it a breaking change
// TODO fix this
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
if (!this.saxStream.write(data, encoding)) {
this.saxStream.once('drain', cb);
}
else {
process.nextTick(cb);
}
}
catch (error) {
callback(error);
}
}
err(msg) {
if (!this.error)
this.error = new Error(msg);
}
}
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
const { createReadStream } = require('fs')
const { parseSitemapIndex, createSitemap } = require('sitemap')
parseSitemapIndex(createReadStream('./example-index.xml')).then(
// produces the same xml
// you can, of course, more practically modify it or store it
(xmlConfig) => console.log(createSitemap(xmlConfig).toString()),
(err) => console.log(err)
)
```
@param {Readable} xml what to parse
@param {number} maxEntries Maximum number of sitemap entries to parse (default: 50,000 per sitemaps.org spec)
@return {Promise<IndexItem[]>} resolves with list of index items that can be fed into a SitemapIndexStream. Rejects with an Error object.
*/
export async function parseSitemapIndex(xml, maxEntries = LIMITS.MAX_SITEMAP_ITEM_LIMIT) {
const urls = [];
return new Promise((resolve, reject) => {
let settled = false;
const parser = new XMLToSitemapIndexStream();
// Handle source stream errors (prevents unhandled error events on xml)
xml.on('error', (error) => {
if (!settled) {
settled = true;
reject(error);
}
});
xml
.pipe(parser)
.on('data', (smi) => {
if (settled)
return;
// Security: Prevent memory exhaustion by limiting number of entries
if (urls.length >= maxEntries) {
settled = true;
reject(new Error(`Sitemap index exceeds maximum allowed entries (${maxEntries})`));
// Immediately destroy both streams to stop further processing (BB-05)
parser.destroy();
xml.destroy();
return;
}
urls.push(smi);
})
.on('end', () => {
if (!settled) {
settled = true;
resolve(urls);
}
})
.on('error', (error) => {
if (!settled) {
settled = true;
reject(error);
}
});
});
}
const defaultObjectStreamOpts = {
lineSeparated: false,
};
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
export class IndexObjectStreamToJSON extends Transform {
lineSeparated;
firstWritten;
constructor(opts = defaultObjectStreamOpts) {
opts.writableObjectMode = true;
super(opts);
this.lineSeparated = opts.lineSeparated;
this.firstWritten = false;
}
_transform(chunk, encoding, cb) {
if (!this.firstWritten) {
this.firstWritten = true;
if (!this.lineSeparated) {
this.push('[');
}
}
else if (this.lineSeparated) {
this.push('\n');
}
else {
this.push(',');
}
if (chunk) {
this.push(JSON.stringify(chunk));
}
cb();
}
_flush(cb) {
if (!this.lineSeparated) {
this.push(']');
}
cb();
}
}
node_modules/sitemap/dist/esm/lib/sitemap-index-stream.d.ts
Generated Vendored
+169
View File
@@ -0,0 +1,169 @@
import { WriteStream } from 'node:fs';
import { Transform, TransformOptions, TransformCallback } from 'node:stream';
import { IndexItem, SitemapItemLoose, ErrorLevel, IndexTagNames } from './types.js';
import { SitemapStream } from './sitemap-stream.js';
export { IndexTagNames };
/**
* Options for the SitemapIndexStream
*/
export interface SitemapIndexStreamOptions extends TransformOptions {
/**
* Whether to output the lastmod date only (no time)
*
* @default false
*/
lastmodDateOnly?: boolean;
/**
* How to handle errors in passed in urls
*
* @default ErrorLevel.WARN
*/
level?: ErrorLevel;
/**
* URL to an XSL stylesheet to include in the XML
*/
xslUrl?: string;
}
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {Transform}
*/
export declare class SitemapIndexStream extends Transform {
lastmodDateOnly: boolean;
level: ErrorLevel;
xslUrl?: string;
private hasHeadOutput;
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapIndexStreamOptions} [opts=defaultStreamOpts] - Stream options.
*/
constructor(opts?: SitemapIndexStreamOptions);
private writeHeadOutput;
_transform(item: IndexItem | string, encoding: string, callback: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
/**
* Callback function type for creating new sitemap streams when the item limit is reached.
*
* This function is called by SitemapAndIndexStream to create a new sitemap file when
* the current one reaches the item limit.
*
* @param i - The zero-based index of the sitemap file being created (0 for first sitemap,
* 1 for second, etc.)
* @returns A tuple containing:
* - [0]: IndexItem or URL string to add to the sitemap index
* - [1]: SitemapStream instance for writing sitemap items
* - [2]: WriteStream where the sitemap will be piped (the stream will be
* awaited for 'finish' before creating the next sitemap)
*
* @example
* ```typescript
* const getSitemapStream = (i: number) => {
* const sitemapStream = new SitemapStream();
* const path = `./sitemap-${i}.xml`;
* const writeStream = createWriteStream(path);
* sitemapStream.pipe(writeStream);
* return [`https://example.com/${path}`, sitemapStream, writeStream];
* };
* ```
*/
type getSitemapStreamFunc = (i: number) => [IndexItem | string, SitemapStream, WriteStream];
/**
* Options for the SitemapAndIndexStream
*
* @extends {SitemapIndexStreamOptions}
*/
export interface SitemapAndIndexStreamOptions extends SitemapIndexStreamOptions {
/**
* Max number of items in each sitemap XML file.
*
* When the limit is reached the current sitemap file will be closed,
* a wait for `finish` on the target write stream will happen,
* and a new sitemap file will be created.
*
* Range: 1 - 50,000
*
* @default 45000
*/
limit?: number;
/**
* Callback for SitemapIndexAndStream that creates a new sitemap stream for a given sitemap index.
*
* Called when a new sitemap file is needed.
*
* The write stream is the destination where the sitemap was piped.
* SitemapAndIndexStream will wait for the `finish` event on each sitemap's
* write stream before moving on to the next sitemap. This ensures that the
* contents of the write stream will be fully written before being used
* by any following operations (e.g. uploading, reading contents for unit tests).
*
* @param i - The index of the sitemap file
* @returns A tuple containing the index item to be written into the sitemap index, the sitemap stream, and the write stream for the sitemap pipe destination
*/
getSitemapStream: getSitemapStreamFunc;
}
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {SitemapIndexStream}
*/
export declare class SitemapAndIndexStream extends SitemapIndexStream {
private itemsWritten;
private getSitemapStream;
private currentSitemap?;
private limit;
private currentSitemapPipeline?;
/**
* Flag to prevent race conditions when creating new sitemap files.
* Set to true while waiting for the current sitemap to finish and
* a new one to be created.
*/
private isCreatingSitemap;
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapAndIndexStreamOptions} opts - Stream options.
*/
constructor(opts: SitemapAndIndexStreamOptions);
_transform(item: SitemapItemLoose, encoding: string, callback: TransformCallback): void;
private writeItem;
/**
* Called when the stream is finished.
* If there is a current sitemap, we wait for it to finish before calling the callback.
* Includes proper event listener cleanup to prevent memory leaks.
*
* @param cb - The callback to invoke when flushing is complete
*/
_flush(cb: TransformCallback): void;
private createSitemap;
}
node_modules/sitemap/dist/esm/lib/sitemap-index-stream.js
Generated Vendored
+359
View File
@@ -0,0 +1,359 @@
import { Transform } from 'node:stream';
import { ErrorLevel, IndexTagNames, } from './types.js';
import { stylesheetInclude } from './sitemap-stream.js';
import { element, otag, ctag } from './sitemap-xml.js';
import { LIMITS, DEFAULT_SITEMAP_ITEM_LIMIT } from './constants.js';
import { validateURL, validateXSLUrl } from './validation.js';
// Re-export IndexTagNames for backward compatibility
export { IndexTagNames };
const xmlDec = '<?xml version="1.0" encoding="UTF-8"?>';
const sitemapIndexTagStart = '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">';
const closetag = '</sitemapindex>';
const defaultStreamOpts = {};
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {Transform}
*/
export class SitemapIndexStream extends Transform {
lastmodDateOnly;
level;
xslUrl;
hasHeadOutput;
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapIndexStreamOptions} [opts=defaultStreamOpts] - Stream options.
*/
constructor(opts = defaultStreamOpts) {
opts.objectMode = true;
super(opts);
this.hasHeadOutput = false;
this.lastmodDateOnly = opts.lastmodDateOnly || false;
this.level = opts.level ?? ErrorLevel.WARN;
if (opts.xslUrl !== undefined) {
validateXSLUrl(opts.xslUrl);
}
this.xslUrl = opts.xslUrl;
}
writeHeadOutput() {
this.hasHeadOutput = true;
let stylesheet = '';
if (this.xslUrl) {
stylesheet = stylesheetInclude(this.xslUrl);
}
this.push(xmlDec + stylesheet + sitemapIndexTagStart);
}
_transform(item, encoding, callback) {
if (!this.hasHeadOutput) {
this.writeHeadOutput();
}
try {
// Validate URL using centralized validation (checks protocol, length, format)
const url = typeof item === 'string' ? item : item.url;
if (!url || typeof url !== 'string') {
const error = new Error('Invalid sitemap index item: URL must be a non-empty string');
if (this.level === ErrorLevel.THROW) {
callback(error);
return;
}
else if (this.level === ErrorLevel.WARN) {
console.warn(error.message, item);
}
// For SILENT or after WARN, skip this item
callback();
return;
}
// Security: Use centralized validation to enforce protocol restrictions,
// length limits, and prevent injection attacks
try {
validateURL(url, 'Sitemap index URL');
}
catch (error) {
// Wrap the validation error with consistent message format
const validationMsg = error instanceof Error ? error.message : String(error);
const err = new Error(`Invalid URL in sitemap index: ${validationMsg}`);
if (this.level === ErrorLevel.THROW) {
callback(err);
return;
}
else if (this.level === ErrorLevel.WARN) {
console.warn(err.message);
}
// For SILENT or after WARN, skip this item
callback();
return;
}
this.push(otag(IndexTagNames.sitemap));
if (typeof item === 'string') {
this.push(element(IndexTagNames.loc, item));
}
else {
this.push(element(IndexTagNames.loc, item.url));
if (item.lastmod) {
try {
const lastmod = new Date(item.lastmod).toISOString();
this.push(element(IndexTagNames.lastmod, this.lastmodDateOnly ? lastmod.slice(0, 10) : lastmod));
}
catch {
const error = new Error(`Invalid lastmod date in sitemap index: ${item.lastmod}`);
if (this.level === ErrorLevel.THROW) {
callback(error);
return;
}
else if (this.level === ErrorLevel.WARN) {
console.warn(error.message);
}
// Continue without lastmod for SILENT or after WARN
}
}
}
this.push(ctag(IndexTagNames.sitemap));
callback();
}
catch (error) {
callback(error instanceof Error ? error : new Error(String(error)));
}
}
_flush(cb) {
if (!this.hasHeadOutput) {
this.writeHeadOutput();
}
this.push(closetag);
cb();
}
}
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {SitemapIndexStream}
*/
export class SitemapAndIndexStream extends SitemapIndexStream {
itemsWritten;
getSitemapStream;
currentSitemap;
limit;
currentSitemapPipeline;
/**
* Flag to prevent race conditions when creating new sitemap files.
* Set to true while waiting for the current sitemap to finish and
* a new one to be created.
*/
isCreatingSitemap;
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapAndIndexStreamOptions} opts - Stream options.
*/
constructor(opts) {
opts.objectMode = true;
super(opts);
this.itemsWritten = 0;
this.getSitemapStream = opts.getSitemapStream;
this.limit = opts.limit ?? DEFAULT_SITEMAP_ITEM_LIMIT;
this.isCreatingSitemap = false;
// Validate limit is within acceptable range per sitemaps.org spec
// See: https://www.sitemaps.org/protocol.html#index
if (this.limit < LIMITS.MIN_SITEMAP_ITEM_LIMIT ||
this.limit > LIMITS.MAX_SITEMAP_ITEM_LIMIT) {
throw new Error(`limit must be between ${LIMITS.MIN_SITEMAP_ITEM_LIMIT} and ${LIMITS.MAX_SITEMAP_ITEM_LIMIT} per sitemaps.org spec, got ${this.limit}`);
}
}
_transform(item, encoding, callback) {
if (this.itemsWritten % this.limit === 0) {
// Prevent race condition if multiple items arrive during sitemap creation
if (this.isCreatingSitemap) {
// Wait and retry on next tick
process.nextTick(() => this._transform(item, encoding, callback));
return;
}
if (this.currentSitemap) {
this.isCreatingSitemap = true;
const currentSitemap = this.currentSitemap;
const currentPipeline = this.currentSitemapPipeline;
// Set up promises with proper cleanup to prevent memory leaks
const onFinish = new Promise((resolve, reject) => {
const finishHandler = () => {
currentSitemap.off('error', errorHandler);
resolve();
};
const errorHandler = (err) => {
currentSitemap.off('finish', finishHandler);
reject(err);
};
currentSitemap.on('finish', finishHandler);
currentSitemap.on('error', errorHandler);
currentSitemap.end();
});
const onPipelineFinish = currentPipeline
? new Promise((resolve, reject) => {
const finishHandler = () => {
currentPipeline.off('error', errorHandler);
resolve();
};
const errorHandler = (err) => {
currentPipeline.off('finish', finishHandler);
reject(err);
};
currentPipeline.on('finish', finishHandler);
currentPipeline.on('error', errorHandler);
})
: Promise.resolve();
Promise.all([onFinish, onPipelineFinish])
.then(() => {
this.isCreatingSitemap = false;
this.createSitemap(encoding);
this.writeItem(item, callback);
})
.catch((err) => {
this.isCreatingSitemap = false;
callback(err);
});
return;
}
else {
this.createSitemap(encoding);
}
}
this.writeItem(item, callback);
}
writeItem(item, callback) {
if (!this.currentSitemap) {
callback(new Error('No sitemap stream available'));
return;
}
if (!this.currentSitemap.write(item)) {
this.currentSitemap.once('drain', callback);
}
else {
process.nextTick(callback);
}
// Increment the count of items written
this.itemsWritten++;
}
/**
* Called when the stream is finished.
* If there is a current sitemap, we wait for it to finish before calling the callback.
* Includes proper event listener cleanup to prevent memory leaks.
*
* @param cb - The callback to invoke when flushing is complete
*/
_flush(cb) {
const currentSitemap = this.currentSitemap;
const currentPipeline = this.currentSitemapPipeline;
const onFinish = new Promise((resolve, reject) => {
if (currentSitemap) {
const finishHandler = () => {
currentSitemap.off('error', errorHandler);
resolve();
};
const errorHandler = (err) => {
currentSitemap.off('finish', finishHandler);
reject(err);
};
currentSitemap.on('finish', finishHandler);
currentSitemap.on('error', errorHandler);
currentSitemap.end();
}
else {
resolve();
}
});
const onPipelineFinish = new Promise((resolve, reject) => {
if (currentPipeline) {
const finishHandler = () => {
currentPipeline.off('error', errorHandler);
resolve();
};
const errorHandler = (err) => {
currentPipeline.off('finish', finishHandler);
reject(err);
};
currentPipeline.on('finish', finishHandler);
currentPipeline.on('error', errorHandler);
// The pipeline (pipe target) will get its end() call
// from the sitemap stream ending.
}
else {
resolve();
}
});
Promise.all([onFinish, onPipelineFinish])
.then(() => {
super._flush(cb);
})
.catch((err) => {
cb(err);
});
}
createSitemap(encoding) {
const sitemapIndex = this.itemsWritten / this.limit;
let result;
try {
result = this.getSitemapStream(sitemapIndex);
}
catch (err) {
this.emit('error', new Error(`getSitemapStream callback threw an error for index ${sitemapIndex}: ${err instanceof Error ? err.message : String(err)}`));
return;
}
// Validate the return value
if (!Array.isArray(result) || result.length !== 3) {
this.emit('error', new Error(`getSitemapStream must return a 3-element array [IndexItem | string, SitemapStream, WriteStream], got: ${typeof result}`));
return;
}
const [idxItem, currentSitemap, currentSitemapPipeline] = result;
// Validate each element
if (!idxItem ||
(typeof idxItem !== 'string' && typeof idxItem !== 'object')) {
this.emit('error', new Error('getSitemapStream must return an IndexItem or string as the first element'));
return;
}
if (!currentSitemap || typeof currentSitemap.write !== 'function') {
this.emit('error', new Error('getSitemapStream must return a SitemapStream as the second element'));
return;
}
if (currentSitemapPipeline &&
typeof currentSitemapPipeline.write !== 'function') {
this.emit('error', new Error('getSitemapStream must return a WriteStream or undefined as the third element'));
return;
}
// Propagate errors from the sitemap stream
currentSitemap.on('error', (err) => this.emit('error', err));
this.currentSitemap = currentSitemap;
this.currentSitemapPipeline = currentSitemapPipeline;
super._transform(idxItem, encoding, () => {
// We are not too concerned about waiting for the index item to be written
// as we'll wait for the file to finish at the end, and index file write
// volume tends to be small in comparison to sitemap writes.
// noop
});
}
}
node_modules/sitemap/dist/esm/lib/sitemap-item-stream.d.ts
Generated Vendored
+21
View File
@@ -0,0 +1,21 @@
import { Transform, TransformOptions, TransformCallback } from 'node:stream';
import { SitemapItem, ErrorLevel } from './types.js';
export interface SitemapItemStreamOptions extends TransformOptions {
level?: ErrorLevel;
}
/**
* Takes a stream of SitemapItemOptions and spits out xml for each
* @example
* // writes <url><loc>https://example.com</loc><url><url><loc>https://example.com/2</loc><url>
* const smis = new SitemapItemStream({level: 'warn'})
* smis.pipe(writestream)
* smis.write({url: 'https://example.com', img: [], video: [], links: []})
* smis.write({url: 'https://example.com/2', img: [], video: [], links: []})
* smis.end()
* @param level - Error level
*/
export declare class SitemapItemStream extends Transform {
level: ErrorLevel;
constructor(opts?: SitemapItemStreamOptions);
_transform(item: SitemapItem, encoding: string, callback: TransformCallback): void;
}
node_modules/sitemap/dist/esm/lib/sitemap-item-stream.js
Generated Vendored
+204
View File
@@ -0,0 +1,204 @@
import { Transform } from 'node:stream';
import { InvalidAttr } from './errors.js';
import { ErrorLevel, TagNames } from './types.js';
import { element, otag, ctag } from './sitemap-xml.js';
/**
* Builds an attributes object for XML elements from configuration object
* Extracts attributes based on colon-delimited keys (e.g., 'price:currency' -> { currency: value })
*
* @param conf - Configuration object containing attribute values
* @param keys - Single key or array of keys in format 'namespace:attribute'
* @returns Record of attribute names to string values (may contain non-string values from conf)
* @throws {InvalidAttr} When key format is invalid (must contain exactly one colon)
*
* @example
* attrBuilder({ 'price:currency': 'USD', 'price:type': 'rent' }, ['price:currency', 'price:type'])
* // Returns: { currency: 'USD', type: 'rent' }
*/
function attrBuilder(conf, keys) {
if (typeof keys === 'string') {
keys = [keys];
}
const iv = {};
return keys.reduce((attrs, key) => {
if (conf[key] !== undefined) {
const keyAr = key.split(':');
if (keyAr.length !== 2) {
throw new InvalidAttr(key);
}
attrs[keyAr[1]] = conf[key];
}
return attrs;
}, iv);
}
/**
* Takes a stream of SitemapItemOptions and spits out xml for each
* @example
* // writes <url><loc>https://example.com</loc><url><url><loc>https://example.com/2</loc><url>
* const smis = new SitemapItemStream({level: 'warn'})
* smis.pipe(writestream)
* smis.write({url: 'https://example.com', img: [], video: [], links: []})
* smis.write({url: 'https://example.com/2', img: [], video: [], links: []})
* smis.end()
* @param level - Error level
*/
export class SitemapItemStream extends Transform {
level;
constructor(opts = { level: ErrorLevel.WARN }) {
opts.objectMode = true;
super(opts);
this.level = opts.level || ErrorLevel.WARN;
}
_transform(item, encoding, callback) {
this.push(otag(TagNames.url));
this.push(element(TagNames.loc, item.url));
if (item.lastmod) {
this.push(element(TagNames.lastmod, item.lastmod));
}
if (item.changefreq) {
this.push(element(TagNames.changefreq, item.changefreq));
}
if (item.priority !== undefined && item.priority !== null) {
if (item.fullPrecisionPriority) {
this.push(element(TagNames.priority, item.priority.toString()));
}
else {
this.push(element(TagNames.priority, item.priority.toFixed(1)));
}
}
item.video.forEach((video) => {
this.push(otag(TagNames['video:video']));
this.push(element(TagNames['video:thumbnail_loc'], video.thumbnail_loc));
this.push(element(TagNames['video:title'], video.title));
this.push(element(TagNames['video:description'], video.description));
if (video.content_loc) {
this.push(element(TagNames['video:content_loc'], video.content_loc));
}
if (video.player_loc) {
this.push(element(TagNames['video:player_loc'], attrBuilder(video, [
'player_loc:autoplay',
'player_loc:allow_embed',
]), video.player_loc));
}
if (video.duration) {
this.push(element(TagNames['video:duration'], video.duration.toString()));
}
if (video.expiration_date) {
this.push(element(TagNames['video:expiration_date'], video.expiration_date));
}
if (video.rating !== undefined) {
this.push(element(TagNames['video:rating'], video.rating.toString()));
}
if (video.view_count !== undefined) {
this.push(element(TagNames['video:view_count'], String(video.view_count)));
}
if (video.publication_date) {
this.push(element(TagNames['video:publication_date'], video.publication_date));
}
if (video.tag && video.tag.length > 0) {
for (const tag of video.tag) {
this.push(element(TagNames['video:tag'], tag));
}
}
if (video.category) {
this.push(element(TagNames['video:category'], video.category));
}
if (video.family_friendly) {
this.push(element(TagNames['video:family_friendly'], video.family_friendly));
}
if (video.restriction) {
this.push(element(TagNames['video:restriction'], attrBuilder(video, 'restriction:relationship'), video.restriction));
}
if (video.gallery_loc) {
this.push(element(TagNames['video:gallery_loc'], attrBuilder(video, 'gallery_loc:title'), video.gallery_loc));
}
if (video.price) {
this.push(element(TagNames['video:price'], attrBuilder(video, [
'price:resolution',
'price:currency',
'price:type',
]), video.price));
}
if (video.requires_subscription) {
this.push(element(TagNames['video:requires_subscription'], video.requires_subscription));
}
if (video.uploader) {
this.push(element(TagNames['video:uploader'], attrBuilder(video, 'uploader:info'), video.uploader));
}
if (video.platform) {
this.push(element(TagNames['video:platform'], attrBuilder(video, 'platform:relationship'), video.platform));
}
if (video.live) {
this.push(element(TagNames['video:live'], video.live));
}
if (video.id) {
this.push(element(TagNames['video:id'], { type: 'url' }, video.id));
}
this.push(ctag(TagNames['video:video']));
});
item.links.forEach((link) => {
this.push(element(TagNames['xhtml:link'], {
rel: 'alternate',
hreflang: link.lang || link.hreflang,
href: link.url,
}));
});
if (item.expires) {
this.push(element(TagNames.expires, new Date(item.expires).toISOString()));
}
if (item.androidLink) {
this.push(element(TagNames['xhtml:link'], {
rel: 'alternate',
href: item.androidLink,
}));
}
if (item.ampLink) {
this.push(element(TagNames['xhtml:link'], {
rel: 'amphtml',
href: item.ampLink,
}));
}
if (item.news) {
this.push(otag(TagNames['news:news']));
this.push(otag(TagNames['news:publication']));
this.push(element(TagNames['news:name'], item.news.publication.name));
this.push(element(TagNames['news:language'], item.news.publication.language));
this.push(ctag(TagNames['news:publication']));
if (item.news.access) {
this.push(element(TagNames['news:access'], item.news.access));
}
if (item.news.genres) {
this.push(element(TagNames['news:genres'], item.news.genres));
}
this.push(element(TagNames['news:publication_date'], item.news.publication_date));
this.push(element(TagNames['news:title'], item.news.title));
if (item.news.keywords) {
this.push(element(TagNames['news:keywords'], item.news.keywords));
}
if (item.news.stock_tickers) {
this.push(element(TagNames['news:stock_tickers'], item.news.stock_tickers));
}
this.push(ctag(TagNames['news:news']));
}
// Image handling
item.img.forEach((image) => {
this.push(otag(TagNames['image:image']));
this.push(element(TagNames['image:loc'], image.url));
if (image.caption) {
this.push(element(TagNames['image:caption'], image.caption));
}
if (image.geoLocation) {
this.push(element(TagNames['image:geo_location'], image.geoLocation));
}
if (image.title) {
this.push(element(TagNames['image:title'], image.title));
}
if (image.license) {
this.push(element(TagNames['image:license'], image.license));
}
this.push(ctag(TagNames['image:image']));
});
this.push(ctag(TagNames.url));
callback();
}
}
node_modules/sitemap/dist/esm/lib/sitemap-parser.d.ts
Generated Vendored
+62
View File
@@ -0,0 +1,62 @@
import type { SAXStream } from 'sax';
import { Readable, Transform, TransformOptions, TransformCallback } from 'node:stream';
import { SitemapItem, ErrorLevel } from './types.js';
type Logger = (level: 'warn' | 'error' | 'info' | 'log', ...message: Parameters<Console['log']>[0]) => void;
export interface XMLToSitemapItemStreamOptions extends TransformOptions {
level?: ErrorLevel;
logger?: Logger | false;
}
/**
* Takes a stream of xml and transforms it into a stream of SitemapItems
* Use this to parse existing sitemaps into config options compatible with this library
*/
export declare class XMLToSitemapItemStream extends Transform {
level: ErrorLevel;
logger: Logger;
/**
* Errors encountered during parsing, capped at LIMITS.MAX_PARSER_ERRORS entries
* to prevent memory DoS from malformed XML (BB-03).
* Use errorCount for the total number of errors regardless of the cap.
*/
errors: Error[];
/** Total number of errors seen, including those beyond the stored cap. */
errorCount: number;
saxStream: SAXStream;
urlCount: number;
constructor(opts?: XMLToSitemapItemStreamOptions);
_transform(data: string, encoding: string, callback: TransformCallback): void;
private err;
}
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
const { createReadStream } = require('fs')
const { parseSitemap, createSitemap } = require('sitemap')
parseSitemap(createReadStream('./example.xml')).then(
// produces the same xml
// you can, of course, more practically modify it or store it
(xmlConfig) => console.log(createSitemap(xmlConfig).toString()),
(err) => console.log(err)
)
```
@param {Readable} xml what to parse
@return {Promise<SitemapItem[]>} resolves with list of sitemap items that can be fed into a SitemapStream. Rejects with an Error object.
*/
export declare function parseSitemap(xml: Readable): Promise<SitemapItem[]>;
export interface ObjectStreamToJSONOptions extends TransformOptions {
lineSeparated: boolean;
}
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
export declare class ObjectStreamToJSON extends Transform {
lineSeparated: boolean;
firstWritten: boolean;
constructor(opts?: ObjectStreamToJSONOptions);
_transform(chunk: SitemapItem, encoding: string, cb: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
export {};
node_modules/sitemap/dist/esm/lib/sitemap-parser.js
Generated Vendored
+779
View File
@@ -0,0 +1,779 @@
import sax from 'sax';
import { Transform, } from 'node:stream';
import { ErrorLevel, TagNames, } from './types.js';
import { isValidChangeFreq, isValidYesNo, isAllowDeny, isPriceType, isResolution, } from './validation.js';
import { LIMITS } from './constants.js';
function isValidTagName(tagName) {
// This only works because the enum name and value are the same
return tagName in TagNames;
}
function getAttrValue(attr) {
if (!attr)
return undefined;
return typeof attr === 'string' ? attr : attr.value;
}
function tagTemplate() {
return {
img: [],
video: [],
links: [],
url: '',
};
}
function videoTemplate() {
return {
tag: [],
thumbnail_loc: '',
title: '',
description: '',
};
}
const imageTemplate = {
url: '',
};
const linkTemplate = {
lang: '',
url: '',
};
function newsTemplate() {
return {
publication: { name: '', language: '' },
publication_date: '',
title: '',
};
}
const defaultLogger = (level, ...message) => console[level](...message);
const defaultStreamOpts = {
logger: defaultLogger,
};
// TODO does this need to end with `options`
/**
* Takes a stream of xml and transforms it into a stream of SitemapItems
* Use this to parse existing sitemaps into config options compatible with this library
*/
export class XMLToSitemapItemStream extends Transform {
level;
logger;
/**
* Errors encountered during parsing, capped at LIMITS.MAX_PARSER_ERRORS entries
* to prevent memory DoS from malformed XML (BB-03).
* Use errorCount for the total number of errors regardless of the cap.
*/
errors;
/** Total number of errors seen, including those beyond the stored cap. */
errorCount;
saxStream;
urlCount;
constructor(opts = defaultStreamOpts) {
opts.objectMode = true;
super(opts);
this.errors = [];
this.errorCount = 0;
this.urlCount = 0;
this.saxStream = sax.createStream(true, {
xmlns: true,
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
strictEntities: true,
trim: true,
});
this.level = opts.level || ErrorLevel.WARN;
if (this.level !== ErrorLevel.SILENT && opts.logger !== false) {
this.logger = opts.logger ?? defaultLogger;
}
else {
this.logger = () => undefined;
}
let currentItem = tagTemplate();
let currentTag;
let currentVideo = videoTemplate();
let currentImage = { ...imageTemplate };
let currentLink = { ...linkTemplate };
let dontpushCurrentLink = false;
this.saxStream.on('opentagstart', (tag) => {
currentTag = tag.name;
if (currentTag.startsWith('news:') && !currentItem.news) {
currentItem.news = newsTemplate();
}
});
this.saxStream.on('opentag', (tag) => {
if (isValidTagName(tag.name)) {
if (tag.name === 'xhtml:link') {
// SAX returns attributes as objects with {name, value, prefix, local, uri}
// Check if required attributes exist and have values
const rel = getAttrValue(tag.attributes.rel);
const href = getAttrValue(tag.attributes.href);
const hreflang = getAttrValue(tag.attributes.hreflang);
if (!rel || !href) {
this.logger('warn', 'xhtml:link missing required rel or href attribute');
this.err('xhtml:link missing required rel or href attribute');
return;
}
if (rel === 'alternate' && hreflang) {
currentLink.url = href;
currentLink.lang = hreflang;
}
else if (rel === 'alternate') {
dontpushCurrentLink = true;
currentItem.androidLink = href;
}
else if (rel === 'amphtml') {
dontpushCurrentLink = true;
currentItem.ampLink = href;
}
else {
this.logger('log', 'unhandled attr for xhtml:link', tag.attributes);
this.err(`unhandled attr for xhtml:link ${JSON.stringify(tag.attributes)}`);
}
}
}
else {
this.logger('warn', 'unhandled tag', tag.name);
this.err(`unhandled tag: ${tag.name}`);
}
});
this.saxStream.on('text', (text) => {
switch (currentTag) {
case 'mobile:mobile':
break;
case TagNames.loc:
// Validate URL
if (text.length > LIMITS.MAX_URL_LENGTH) {
this.logger('warn', `URL exceeds max length of ${LIMITS.MAX_URL_LENGTH}: ${text.substring(0, 100)}...`);
this.err(`URL exceeds max length of ${LIMITS.MAX_URL_LENGTH}`);
}
else if (!LIMITS.URL_PROTOCOL_REGEX.test(text)) {
this.logger('warn', `URL must start with http:// or https://: ${text}`);
this.err(`URL must start with http:// or https://: ${text}`);
}
else {
currentItem.url = text;
}
break;
case TagNames.changefreq:
if (isValidChangeFreq(text)) {
currentItem.changefreq = text;
}
break;
case TagNames.priority:
{
const priority = parseFloat(text);
if (isNaN(priority) ||
!isFinite(priority) ||
priority < 0 ||
priority > 1) {
this.logger('warn', `Invalid priority "${text}" - must be between 0 and 1`);
this.err(`Invalid priority "${text}" - must be between 0 and 1`);
}
else {
currentItem.priority = priority;
}
}
break;
case TagNames.lastmod:
if (LIMITS.ISO_DATE_REGEX.test(text)) {
currentItem.lastmod = text;
}
else {
this.logger('warn', `Invalid lastmod date format "${text}" - expected ISO 8601 format`);
this.err(`Invalid lastmod date format "${text}" - expected ISO 8601 format`);
}
break;
case TagNames['video:thumbnail_loc']:
currentVideo.thumbnail_loc = text;
break;
case TagNames['video:tag']:
if (currentVideo.tag.length < LIMITS.MAX_TAGS_PER_VIDEO) {
currentVideo.tag.push(text);
}
else {
this.logger('warn', `video has too many tags (max ${LIMITS.MAX_TAGS_PER_VIDEO})`);
this.err(`video has too many tags (max ${LIMITS.MAX_TAGS_PER_VIDEO})`);
}
break;
case TagNames['video:duration']:
{
const duration = parseInt(text, 10);
if (isNaN(duration) ||
!isFinite(duration) ||
duration < 0 ||
duration > 28800) {
this.logger('warn', `Invalid video duration "${text}" - must be between 0 and 28800 seconds`);
this.err(`Invalid video duration "${text}" - must be between 0 and 28800 seconds`);
}
else {
currentVideo.duration = duration;
}
}
break;
case TagNames['video:player_loc']:
currentVideo.player_loc = text;
break;
case TagNames['video:content_loc']:
currentVideo.content_loc = text;
break;
case TagNames['video:requires_subscription']:
if (isValidYesNo(text)) {
currentVideo.requires_subscription = text;
}
break;
case TagNames['video:publication_date']:
if (LIMITS.ISO_DATE_REGEX.test(text)) {
currentVideo.publication_date = text;
}
else {
this.logger('warn', `Invalid video publication_date format "${text}" - expected ISO 8601 format`);
this.err(`Invalid video publication_date format "${text}" - expected ISO 8601 format`);
}
break;
case TagNames['video:id']:
currentVideo.id = text;
break;
case TagNames['video:restriction']:
currentVideo.restriction = text;
break;
case TagNames['video:view_count']:
{
const viewCount = parseInt(text, 10);
if (isNaN(viewCount) || !isFinite(viewCount) || viewCount < 0) {
this.logger('warn', `Invalid video view_count "${text}" - must be a positive integer`);
this.err(`Invalid video view_count "${text}" - must be a positive integer`);
}
else {
currentVideo.view_count = viewCount;
}
}
break;
case TagNames['video:uploader']:
currentVideo.uploader = text;
break;
case TagNames['video:family_friendly']:
if (isValidYesNo(text)) {
currentVideo.family_friendly = text;
}
break;
case TagNames['video:expiration_date']:
if (LIMITS.ISO_DATE_REGEX.test(text)) {
currentVideo.expiration_date = text;
}
else {
this.logger('warn', `Invalid video expiration_date format "${text}" - expected ISO 8601 format`);
this.err(`Invalid video expiration_date format "${text}" - expected ISO 8601 format`);
}
break;
case TagNames['video:platform']:
currentVideo.platform = text;
break;
case TagNames['video:price']:
currentVideo.price = text;
break;
case TagNames['video:rating']:
{
const rating = parseFloat(text);
if (isNaN(rating) ||
!isFinite(rating) ||
rating < 0 ||
rating > 5) {
this.logger('warn', `Invalid video rating "${text}" - must be between 0 and 5`);
this.err(`Invalid video rating "${text}" - must be between 0 and 5`);
}
else {
currentVideo.rating = rating;
}
}
break;
case TagNames['video:category']:
currentVideo.category = text;
break;
case TagNames['video:live']:
if (isValidYesNo(text)) {
currentVideo.live = text;
}
break;
case TagNames['video:gallery_loc']:
currentVideo.gallery_loc = text;
break;
case TagNames['image:loc']:
currentImage.url = text;
break;
case TagNames['image:geo_location']:
currentImage.geoLocation = text;
break;
case TagNames['image:license']:
currentImage.license = text;
break;
case TagNames['news:access']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (text === 'Registration' || text === 'Subscription') {
currentItem.news.access = text;
}
else {
this.logger('warn', `Invalid news:access value "${text}" - must be "Registration" or "Subscription"`);
this.err(`Invalid news:access value "${text}" - must be "Registration" or "Subscription"`);
}
break;
case TagNames['news:genres']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.genres = text;
break;
case TagNames['news:publication_date']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (LIMITS.ISO_DATE_REGEX.test(text)) {
currentItem.news.publication_date = text;
}
else {
this.logger('warn', `Invalid news publication_date format "${text}" - expected ISO 8601 format`);
this.err(`Invalid news publication_date format "${text}" - expected ISO 8601 format`);
}
break;
case TagNames['news:keywords']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.keywords = text;
break;
case TagNames['news:stock_tickers']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.stock_tickers = text;
break;
case TagNames['news:language']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.publication.language = text;
break;
case TagNames['video:title']:
if (currentVideo.title.length + text.length <=
LIMITS.MAX_VIDEO_TITLE_LENGTH) {
currentVideo.title += text;
}
else {
this.logger('warn', `video title exceeds max length of ${LIMITS.MAX_VIDEO_TITLE_LENGTH}`);
this.err(`video title exceeds max length of ${LIMITS.MAX_VIDEO_TITLE_LENGTH}`);
}
break;
case TagNames['video:description']:
if (currentVideo.description.length + text.length <=
LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH) {
currentVideo.description += text;
}
else {
this.logger('warn', `video description exceeds max length of ${LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH}`);
this.err(`video description exceeds max length of ${LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH}`);
}
break;
case TagNames['news:name']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (currentItem.news.publication.name.length + text.length <=
LIMITS.MAX_NEWS_NAME_LENGTH) {
currentItem.news.publication.name += text;
}
else {
this.logger('warn', `news name exceeds max length of ${LIMITS.MAX_NEWS_NAME_LENGTH}`);
this.err(`news name exceeds max length of ${LIMITS.MAX_NEWS_NAME_LENGTH}`);
}
break;
case TagNames['news:title']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (currentItem.news.title.length + text.length <=
LIMITS.MAX_NEWS_TITLE_LENGTH) {
currentItem.news.title += text;
}
else {
this.logger('warn', `news title exceeds max length of ${LIMITS.MAX_NEWS_TITLE_LENGTH}`);
this.err(`news title exceeds max length of ${LIMITS.MAX_NEWS_TITLE_LENGTH}`);
}
break;
case TagNames['image:caption']:
if (!currentImage.caption) {
currentImage.caption =
text.length <= LIMITS.MAX_IMAGE_CAPTION_LENGTH
? text
: text.substring(0, LIMITS.MAX_IMAGE_CAPTION_LENGTH);
if (text.length > LIMITS.MAX_IMAGE_CAPTION_LENGTH) {
this.logger('warn', `image caption exceeds max length of ${LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
this.err(`image caption exceeds max length of ${LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
}
}
else if (currentImage.caption.length + text.length <=
LIMITS.MAX_IMAGE_CAPTION_LENGTH) {
currentImage.caption += text;
}
else {
this.logger('warn', `image caption exceeds max length of ${LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
this.err(`image caption exceeds max length of ${LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
}
break;
case TagNames['image:title']:
if (!currentImage.title) {
currentImage.title =
text.length <= LIMITS.MAX_IMAGE_TITLE_LENGTH
? text
: text.substring(0, LIMITS.MAX_IMAGE_TITLE_LENGTH);
if (text.length > LIMITS.MAX_IMAGE_TITLE_LENGTH) {
this.logger('warn', `image title exceeds max length of ${LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
this.err(`image title exceeds max length of ${LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
}
}
else if (currentImage.title.length + text.length <=
LIMITS.MAX_IMAGE_TITLE_LENGTH) {
currentImage.title += text;
}
else {
this.logger('warn', `image title exceeds max length of ${LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
this.err(`image title exceeds max length of ${LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
}
break;
default:
this.logger('log', 'unhandled text for tag:', currentTag, `'${text}'`);
this.err(`unhandled text for tag: ${currentTag} '${text}'`);
break;
}
});
this.saxStream.on('cdata', (text) => {
switch (currentTag) {
case TagNames.loc:
// Validate URL
if (text.length > LIMITS.MAX_URL_LENGTH) {
this.logger('warn', `URL exceeds max length of ${LIMITS.MAX_URL_LENGTH}: ${text.substring(0, 100)}...`);
this.err(`URL exceeds max length of ${LIMITS.MAX_URL_LENGTH}`);
}
else if (!LIMITS.URL_PROTOCOL_REGEX.test(text)) {
this.logger('warn', `URL must start with http:// or https://: ${text}`);
this.err(`URL must start with http:// or https://: ${text}`);
}
else {
currentItem.url = text;
}
break;
case TagNames['image:loc']:
currentImage.url = text;
break;
case TagNames['video:title']:
if (currentVideo.title.length + text.length <=
LIMITS.MAX_VIDEO_TITLE_LENGTH) {
currentVideo.title += text;
}
else {
this.logger('warn', `video title exceeds max length of ${LIMITS.MAX_VIDEO_TITLE_LENGTH}`);
this.err(`video title exceeds max length of ${LIMITS.MAX_VIDEO_TITLE_LENGTH}`);
}
break;
case TagNames['video:description']:
if (currentVideo.description.length + text.length <=
LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH) {
currentVideo.description += text;
}
else {
this.logger('warn', `video description exceeds max length of ${LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH}`);
this.err(`video description exceeds max length of ${LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH}`);
}
break;
case TagNames['news:name']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (currentItem.news.publication.name.length + text.length <=
LIMITS.MAX_NEWS_NAME_LENGTH) {
currentItem.news.publication.name += text;
}
else {
this.logger('warn', `news name exceeds max length of ${LIMITS.MAX_NEWS_NAME_LENGTH}`);
this.err(`news name exceeds max length of ${LIMITS.MAX_NEWS_NAME_LENGTH}`);
}
break;
case TagNames['news:title']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
if (currentItem.news.title.length + text.length <=
LIMITS.MAX_NEWS_TITLE_LENGTH) {
currentItem.news.title += text;
}
else {
this.logger('warn', `news title exceeds max length of ${LIMITS.MAX_NEWS_TITLE_LENGTH}`);
this.err(`news title exceeds max length of ${LIMITS.MAX_NEWS_TITLE_LENGTH}`);
}
break;
case TagNames['image:caption']:
if (!currentImage.caption) {
currentImage.caption =
text.length <= LIMITS.MAX_IMAGE_CAPTION_LENGTH
? text
: text.substring(0, LIMITS.MAX_IMAGE_CAPTION_LENGTH);
if (text.length > LIMITS.MAX_IMAGE_CAPTION_LENGTH) {
this.logger('warn', `image caption exceeds max length of ${LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
this.err(`image caption exceeds max length of ${LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
}
}
else if (currentImage.caption.length + text.length <=
LIMITS.MAX_IMAGE_CAPTION_LENGTH) {
currentImage.caption += text;
}
else {
this.logger('warn', `image caption exceeds max length of ${LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
this.err(`image caption exceeds max length of ${LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
}
break;
case TagNames['image:title']:
if (!currentImage.title) {
currentImage.title =
text.length <= LIMITS.MAX_IMAGE_TITLE_LENGTH
? text
: text.substring(0, LIMITS.MAX_IMAGE_TITLE_LENGTH);
if (text.length > LIMITS.MAX_IMAGE_TITLE_LENGTH) {
this.logger('warn', `image title exceeds max length of ${LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
this.err(`image title exceeds max length of ${LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
}
}
else if (currentImage.title.length + text.length <=
LIMITS.MAX_IMAGE_TITLE_LENGTH) {
currentImage.title += text;
}
else {
this.logger('warn', `image title exceeds max length of ${LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
this.err(`image title exceeds max length of ${LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
}
break;
default:
this.logger('log', 'unhandled cdata for tag:', currentTag);
this.err(`unhandled cdata for tag: ${currentTag}`);
break;
}
});
this.saxStream.on('attribute', (attr) => {
switch (currentTag) {
case TagNames['urlset']:
case TagNames['xhtml:link']:
case TagNames['video:id']:
break;
case TagNames['video:restriction']:
if (attr.name === 'relationship' && isAllowDeny(attr.value)) {
currentVideo['restriction:relationship'] = attr.value;
}
else {
this.logger('log', 'unhandled attr', currentTag, attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case TagNames['video:price']:
if (attr.name === 'type' && isPriceType(attr.value)) {
currentVideo['price:type'] = attr.value;
}
else if (attr.name === 'currency') {
currentVideo['price:currency'] = attr.value;
}
else if (attr.name === 'resolution' && isResolution(attr.value)) {
currentVideo['price:resolution'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:price', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case TagNames['video:player_loc']:
if (attr.name === 'autoplay') {
currentVideo['player_loc:autoplay'] = attr.value;
}
else if (attr.name === 'allow_embed' && isValidYesNo(attr.value)) {
currentVideo['player_loc:allow_embed'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:player_loc', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case TagNames['video:platform']:
if (attr.name === 'relationship' && isAllowDeny(attr.value)) {
currentVideo['platform:relationship'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:platform', attr.name, attr.value);
this.err(`unhandled attr: ${currentTag} ${attr.name} ${attr.value}`);
}
break;
case TagNames['video:gallery_loc']:
if (attr.name === 'title') {
currentVideo['gallery_loc:title'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:galler_loc', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case TagNames['video:uploader']:
if (attr.name === 'info') {
currentVideo['uploader:info'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:uploader', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
default:
this.logger('log', 'unhandled attr', currentTag, attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
});
this.saxStream.on('closetag', (tag) => {
switch (tag) {
case TagNames.url:
this.urlCount++;
if (this.urlCount > LIMITS.MAX_URL_ENTRIES) {
this.logger('error', `Sitemap exceeds maximum of ${LIMITS.MAX_URL_ENTRIES} URLs`);
this.err(`Sitemap exceeds maximum of ${LIMITS.MAX_URL_ENTRIES} URLs`);
currentItem = tagTemplate();
break;
}
this.push(currentItem);
currentItem = tagTemplate();
break;
case TagNames['video:video']:
if (currentItem.video.length < LIMITS.MAX_VIDEOS_PER_URL) {
currentItem.video.push(currentVideo);
}
else {
this.logger('warn', `URL has too many videos (max ${LIMITS.MAX_VIDEOS_PER_URL})`);
this.err(`URL has too many videos (max ${LIMITS.MAX_VIDEOS_PER_URL})`);
}
currentVideo = videoTemplate();
break;
case TagNames['image:image']:
if (currentItem.img.length < LIMITS.MAX_IMAGES_PER_URL) {
currentItem.img.push(currentImage);
}
else {
this.logger('warn', `URL has too many images (max ${LIMITS.MAX_IMAGES_PER_URL})`);
this.err(`URL has too many images (max ${LIMITS.MAX_IMAGES_PER_URL})`);
}
currentImage = { ...imageTemplate };
break;
case TagNames['xhtml:link']:
if (!dontpushCurrentLink) {
if (currentItem.links.length < LIMITS.MAX_LINKS_PER_URL) {
currentItem.links.push(currentLink);
}
else {
this.logger('warn', `URL has too many links (max ${LIMITS.MAX_LINKS_PER_URL})`);
this.err(`URL has too many links (max ${LIMITS.MAX_LINKS_PER_URL})`);
}
}
currentLink = { ...linkTemplate };
dontpushCurrentLink = false; // Reset flag for next link
break;
default:
break;
}
});
}
_transform(data, encoding, callback) {
try {
const cb = () => callback(this.level === ErrorLevel.THROW && this.errors.length > 0
? this.errors[0]
: null);
// correcting the type here can be done without making it a breaking change
// TODO fix this
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
if (!this.saxStream.write(data, encoding)) {
this.saxStream.once('drain', cb);
}
else {
process.nextTick(cb);
}
}
catch (error) {
callback(error);
}
}
err(msg) {
this.errorCount++;
if (this.errors.length < LIMITS.MAX_PARSER_ERRORS) {
this.errors.push(new Error(msg));
}
}
}
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
const { createReadStream } = require('fs')
const { parseSitemap, createSitemap } = require('sitemap')
parseSitemap(createReadStream('./example.xml')).then(
// produces the same xml
// you can, of course, more practically modify it or store it
(xmlConfig) => console.log(createSitemap(xmlConfig).toString()),
(err) => console.log(err)
)
```
@param {Readable} xml what to parse
@return {Promise<SitemapItem[]>} resolves with list of sitemap items that can be fed into a SitemapStream. Rejects with an Error object.
*/
export async function parseSitemap(xml) {
const urls = [];
return new Promise((resolve, reject) => {
xml
.pipe(new XMLToSitemapItemStream())
.on('data', (smi) => urls.push(smi))
.on('end', () => {
resolve(urls);
})
.on('error', (error) => {
reject(error);
});
});
}
const defaultObjectStreamOpts = {
lineSeparated: false,
};
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
export class ObjectStreamToJSON extends Transform {
lineSeparated;
firstWritten;
constructor(opts = defaultObjectStreamOpts) {
opts.writableObjectMode = true;
super(opts);
this.lineSeparated = opts.lineSeparated;
this.firstWritten = false;
}
_transform(chunk, encoding, cb) {
if (!this.firstWritten) {
this.firstWritten = true;
if (!this.lineSeparated) {
this.push('[');
}
}
else if (this.lineSeparated) {
this.push('\n');
}
else {
this.push(',');
}
if (chunk) {
this.push(JSON.stringify(chunk));
}
cb();
}
_flush(cb) {
if (!this.lineSeparated) {
this.push(']');
}
cb();
}
}
node_modules/sitemap/dist/esm/lib/sitemap-simple.d.ts
Generated Vendored
+63
View File
@@ -0,0 +1,63 @@
import { Readable } from 'node:stream';
import { SitemapItemLoose } from './types.js';
/**
* Options for the simpleSitemapAndIndex function
*/
export interface SimpleSitemapAndIndexOptions {
/**
* The hostname for all URLs
* Must be a valid http:// or https:// URL
*/
hostname: string;
/**
* The hostname for the sitemaps if different than hostname
* Must be a valid http:// or https:// URL
*/
sitemapHostname?: string;
/**
* The urls you want to make a sitemap out of.
* Can be an array of items, a file path string, a Readable stream, or an array of strings
*/
sourceData: SitemapItemLoose[] | string | Readable | string[];
/**
* Where to write the sitemaps and index
* Must be a relative path without path traversal sequences
*/
destinationDir: string;
/**
* Where the sitemaps are relative to the hostname. Defaults to root.
* Must not contain path traversal sequences
*/
publicBasePath?: string;
/**
* How many URLs to write before switching to a new file
* Must be between 1 and 50,000 per sitemaps.org spec
* @default 50000
*/
limit?: number;
/**
* Whether to compress the written files
* @default true
*/
gzip?: boolean;
/**
* Optional URL to an XSL stylesheet
* Must be a valid http:// or https:// URL
*/
xslUrl?: string;
}
/**
* A simpler interface for creating sitemaps and indexes.
* Automatically handles splitting large datasets into multiple sitemap files.
*
* @param options - Configuration options
* @returns A promise that resolves when all sitemaps and the index are written
* @throws {InvalidHostnameError} If hostname or sitemapHostname is invalid
* @throws {InvalidPathError} If destinationDir contains path traversal
* @throws {InvalidPublicBasePathError} If publicBasePath is invalid
* @throws {InvalidLimitError} If limit is out of range
* @throws {InvalidXSLUrlError} If xslUrl is invalid
* @throws {Error} If sourceData type is not supported
*/
export declare const simpleSitemapAndIndex: ({ hostname, sitemapHostname, sourceData, destinationDir, limit, gzip, publicBasePath, xslUrl, }: SimpleSitemapAndIndexOptions) => Promise<void>;
export default simpleSitemapAndIndex;
node_modules/sitemap/dist/esm/lib/sitemap-simple.js
Generated Vendored
+109
View File
@@ -0,0 +1,109 @@
import { SitemapAndIndexStream } from './sitemap-index-stream.js';
import { SitemapStream } from './sitemap-stream.js';
import { lineSeparatedURLsToSitemapOptions } from './utils.js';
import { createGzip } from 'node:zlib';
import { createWriteStream, createReadStream, promises, } from 'node:fs';
import { normalize, resolve } from 'node:path';
import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import { URL } from 'node:url';
import { validateURL, validatePath, validateLimit, validatePublicBasePath, validateXSLUrl, } from './validation.js';
/**
* A simpler interface for creating sitemaps and indexes.
* Automatically handles splitting large datasets into multiple sitemap files.
*
* @param options - Configuration options
* @returns A promise that resolves when all sitemaps and the index are written
* @throws {InvalidHostnameError} If hostname or sitemapHostname is invalid
* @throws {InvalidPathError} If destinationDir contains path traversal
* @throws {InvalidPublicBasePathError} If publicBasePath is invalid
* @throws {InvalidLimitError} If limit is out of range
* @throws {InvalidXSLUrlError} If xslUrl is invalid
* @throws {Error} If sourceData type is not supported
*/
export const simpleSitemapAndIndex = async ({ hostname, sitemapHostname = hostname, // if different
sourceData, destinationDir, limit = 50000, gzip = true, publicBasePath = './', xslUrl, }) => {
// Validate all inputs upfront
validateURL(hostname, 'hostname');
validateURL(sitemapHostname, 'sitemapHostname');
validatePath(destinationDir, 'destinationDir');
validateLimit(limit);
validatePublicBasePath(publicBasePath);
if (xslUrl) {
validateXSLUrl(xslUrl);
}
// Create destination directory with error context
try {
await promises.mkdir(destinationDir, { recursive: true });
}
catch (err) {
throw new Error(`Failed to create destination directory "${destinationDir}": ${err instanceof Error ? err.message : String(err)}`);
}
// Normalize publicBasePath (don't mutate the parameter)
const normalizedPublicBasePath = publicBasePath.endsWith('/')
? publicBasePath
: publicBasePath + '/';
const sitemapAndIndexStream = new SitemapAndIndexStream({
limit,
getSitemapStream: (i) => {
const sitemapStream = new SitemapStream({
hostname,
xslUrl,
});
const path = `./sitemap-${i}.xml`;
const writePath = resolve(destinationDir, path + (gzip ? '.gz' : ''));
// Construct public path for the sitemap index
const publicPath = normalize(normalizedPublicBasePath + path);
// Construct the URL with proper error handling
let sitemapUrl;
try {
sitemapUrl = new URL(`${publicPath}${gzip ? '.gz' : ''}`, sitemapHostname).toString();
}
catch (err) {
throw new Error(`Failed to construct sitemap URL for index ${i}: ${err instanceof Error ? err.message : String(err)}`);
}
let writeStream;
if (gzip) {
writeStream = sitemapStream
.pipe(createGzip()) // compress the output of the sitemap
.pipe(createWriteStream(writePath)); // write it to sitemap-NUMBER.xml
}
else {
writeStream = sitemapStream.pipe(createWriteStream(writePath)); // write it to sitemap-NUMBER.xml
}
return [sitemapUrl, sitemapStream, writeStream];
},
});
// Handle different sourceData types with proper error handling
let src;
if (typeof sourceData === 'string') {
try {
src = lineSeparatedURLsToSitemapOptions(createReadStream(sourceData));
}
catch (err) {
throw new Error(`Failed to read sourceData file "${sourceData}": ${err instanceof Error ? err.message : String(err)}`);
}
}
else if (sourceData instanceof Readable) {
src = sourceData;
}
else if (Array.isArray(sourceData)) {
src = Readable.from(sourceData);
}
else {
throw new Error(`Invalid sourceData type: expected array, string (file path), or Readable stream, got ${typeof sourceData}`);
}
const writePath = resolve(destinationDir, `./sitemap-index.xml${gzip ? '.gz' : ''}`);
try {
if (gzip) {
return await pipeline(src, sitemapAndIndexStream, createGzip(), createWriteStream(writePath));
}
else {
return await pipeline(src, sitemapAndIndexStream, createWriteStream(writePath));
}
}
catch (err) {
throw new Error(`Failed to write sitemap files: ${err instanceof Error ? err.message : String(err)}`);
}
};
export default simpleSitemapAndIndex;
node_modules/sitemap/dist/esm/lib/sitemap-stream.d.ts
Generated Vendored
+79
View File
@@ -0,0 +1,79 @@
import { Transform, TransformOptions, TransformCallback, Readable } from 'node:stream';
import { SitemapItemLoose, ErrorLevel, ErrorHandler } from './types.js';
export declare const stylesheetInclude: (url: string) => string;
export interface NSArgs {
news: boolean;
video: boolean;
xhtml: boolean;
image: boolean;
custom?: string[];
}
export declare const closetag = "</urlset>";
export interface SitemapStreamOptions extends TransformOptions {
hostname?: string;
level?: ErrorLevel;
lastmodDateOnly?: boolean;
xmlns?: NSArgs;
xslUrl?: string;
errorHandler?: ErrorHandler;
}
/**
* A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream)
* for turning a
* [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams)
* of either [SitemapItemOptions](#sitemap-item-options) or url strings into a
* Sitemap. The readable stream it transforms **must** be in object mode.
*
* @param {SitemapStreamOptions} opts - Configuration options
* @param {string} [opts.hostname] - Base URL for relative paths. Must use http:// or https:// protocol
* @param {ErrorLevel} [opts.level=ErrorLevel.WARN] - Error handling level (SILENT, WARN, or THROW)
* @param {boolean} [opts.lastmodDateOnly=false] - Format lastmod as date only (YYYY-MM-DD)
* @param {NSArgs} [opts.xmlns] - Control which XML namespaces to include in output
* @param {string} [opts.xslUrl] - URL to XSL stylesheet for sitemap display. Must use http:// or https://
* @param {ErrorHandler} [opts.errorHandler] - Custom error handler function
*
* @throws {InvalidHostnameError} If hostname is provided but invalid (non-http(s), malformed, or >2048 chars)
* @throws {InvalidXSLUrlError} If xslUrl is provided but invalid (non-http(s), malformed, >2048 chars, or contains malicious content)
* @throws {Error} If xmlns.custom contains invalid namespace declarations
*
* @example
* ```typescript
* const stream = new SitemapStream({
* hostname: 'https://example.com',
* level: ErrorLevel.THROW
* });
* stream.write({ url: '/page', changefreq: 'daily' });
* stream.end();
* ```
*
* @security
* - Hostname and xslUrl are validated to prevent URL injection attacks
* - Custom namespaces are validated to prevent XML injection
* - All URLs are normalized and validated before output
* - XML content is properly escaped to prevent injection
*/
export declare class SitemapStream extends Transform {
hostname?: string;
level: ErrorLevel;
hasHeadOutput: boolean;
xmlNS: NSArgs;
xslUrl?: string;
errorHandler?: ErrorHandler;
private smiStream;
lastmodDateOnly: boolean;
constructor(opts?: SitemapStreamOptions);
_transform(item: SitemapItemLoose, encoding: string, callback: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
/**
* Converts a readable stream into a promise that resolves with the concatenated data from the stream.
*
* The function listens for 'data' events from the stream, and when the stream ends, it resolves the promise with the concatenated data. If an error occurs while reading from the stream, the promise is rejected with the error.
*
* ⚠️ CAUTION: This function should not generally be used in production / when writing to files as it holds a copy of the entire file contents in memory until finished.
*
* @param {Readable} stream - The readable stream to convert to a promise.
* @returns {Promise<Buffer>} A promise that resolves with the concatenated data from the stream as a Buffer, or rejects with an error if one occurred while reading from the stream. If the stream is empty, the promise is rejected with an EmptyStream error.
* @throws {EmptyStream} If the stream is empty.
*/
export declare function streamToPromise(stream: Readable): Promise<Buffer>;
node_modules/sitemap/dist/esm/lib/sitemap-stream.js
Generated Vendored
+212
View File
@@ -0,0 +1,212 @@
import { Transform, Writable, } from 'node:stream';
import { ErrorLevel } from './types.js';
import { normalizeURL } from './utils.js';
import { validateSMIOptions, validateURL, validateXSLUrl, } from './validation.js';
import { SitemapItemStream } from './sitemap-item-stream.js';
import { EmptyStream, EmptySitemap } from './errors.js';
import { LIMITS } from './constants.js';
const xmlDec = '<?xml version="1.0" encoding="UTF-8"?>';
export const stylesheetInclude = (url) => {
const safe = url
.replace(/&/g, '&amp;')
.replace(/"/g, '&quot;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
return `<?xml-stylesheet type="text/xsl" href="${safe}"?>`;
};
const urlsetTagStart = '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"';
/**
* Validates custom namespace declarations for security
* @param custom - Array of custom namespace declarations
* @throws {Error} If namespace format is invalid or contains malicious content
*/
function validateCustomNamespaces(custom) {
if (!Array.isArray(custom)) {
throw new Error('Custom namespaces must be an array');
}
// Limit number of custom namespaces to prevent DoS
if (custom.length > LIMITS.MAX_CUSTOM_NAMESPACES) {
throw new Error(`Too many custom namespaces: ${custom.length} exceeds limit of ${LIMITS.MAX_CUSTOM_NAMESPACES}`);
}
// Basic format validation for xmlns declarations and namespace-qualified attributes
// Supports both xmlns:prefix="uri" and prefix:attribute="value" (e.g., xsi:schemaLocation)
const xmlAttributePattern = /^[a-zA-Z_][\w.-]*:[a-zA-Z_][\w.-]*="[^"<>]*"$/;
for (const ns of custom) {
if (typeof ns !== 'string' || ns.length === 0) {
throw new Error('Custom namespace must be a non-empty string');
}
if (ns.length > LIMITS.MAX_NAMESPACE_LENGTH) {
throw new Error(`Custom namespace exceeds maximum length of ${LIMITS.MAX_NAMESPACE_LENGTH} characters: ${ns.substring(0, 50)}...`);
}
// Check for potentially malicious content BEFORE format check
// (format check will reject < and > but we want specific error message)
const lowerNs = ns.toLowerCase();
if (lowerNs.includes('<script') ||
lowerNs.includes('javascript:') ||
lowerNs.includes('data:text/html')) {
throw new Error(`Custom namespace contains potentially malicious content: ${ns.substring(0, 50)}`);
}
// Check format matches xmlns declaration or namespace-qualified attribute
if (!xmlAttributePattern.test(ns)) {
throw new Error(`Invalid namespace format (must be prefix:name="value", e.g., xmlns:prefix="uri" or xsi:schemaLocation="..."): ${ns.substring(0, 50)}`);
}
}
}
const getURLSetNs = ({ news, video, image, xhtml, custom }, xslURL) => {
let ns = xmlDec;
if (xslURL) {
ns += stylesheetInclude(xslURL);
}
ns += urlsetTagStart;
if (news) {
ns += ' xmlns:news="http://www.google.com/schemas/sitemap-news/0.9"';
}
if (xhtml) {
ns += ' xmlns:xhtml="http://www.w3.org/1999/xhtml"';
}
if (image) {
ns += ' xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"';
}
if (video) {
ns += ' xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"';
}
if (custom) {
validateCustomNamespaces(custom);
ns += ' ' + custom.join(' ');
}
return ns + '>';
};
export const closetag = '</urlset>';
const defaultXMLNS = {
news: true,
xhtml: true,
image: true,
video: true,
};
const defaultStreamOpts = {
xmlns: defaultXMLNS,
};
/**
* A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream)
* for turning a
* [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams)
* of either [SitemapItemOptions](#sitemap-item-options) or url strings into a
* Sitemap. The readable stream it transforms **must** be in object mode.
*
* @param {SitemapStreamOptions} opts - Configuration options
* @param {string} [opts.hostname] - Base URL for relative paths. Must use http:// or https:// protocol
* @param {ErrorLevel} [opts.level=ErrorLevel.WARN] - Error handling level (SILENT, WARN, or THROW)
* @param {boolean} [opts.lastmodDateOnly=false] - Format lastmod as date only (YYYY-MM-DD)
* @param {NSArgs} [opts.xmlns] - Control which XML namespaces to include in output
* @param {string} [opts.xslUrl] - URL to XSL stylesheet for sitemap display. Must use http:// or https://
* @param {ErrorHandler} [opts.errorHandler] - Custom error handler function
*
* @throws {InvalidHostnameError} If hostname is provided but invalid (non-http(s), malformed, or >2048 chars)
* @throws {InvalidXSLUrlError} If xslUrl is provided but invalid (non-http(s), malformed, >2048 chars, or contains malicious content)
* @throws {Error} If xmlns.custom contains invalid namespace declarations
*
* @example
* ```typescript
* const stream = new SitemapStream({
* hostname: 'https://example.com',
* level: ErrorLevel.THROW
* });
* stream.write({ url: '/page', changefreq: 'daily' });
* stream.end();
* ```
*
* @security
* - Hostname and xslUrl are validated to prevent URL injection attacks
* - Custom namespaces are validated to prevent XML injection
* - All URLs are normalized and validated before output
* - XML content is properly escaped to prevent injection
*/
export class SitemapStream extends Transform {
hostname;
level;
hasHeadOutput;
xmlNS;
xslUrl;
errorHandler;
smiStream;
lastmodDateOnly;
constructor(opts = defaultStreamOpts) {
opts.objectMode = true;
super(opts);
// Validate hostname if provided
if (opts.hostname !== undefined) {
validateURL(opts.hostname, 'hostname');
}
// Validate xslUrl if provided
if (opts.xslUrl !== undefined) {
validateXSLUrl(opts.xslUrl);
}
this.hasHeadOutput = false;
this.hostname = opts.hostname;
this.level = opts.level || ErrorLevel.WARN;
this.errorHandler = opts.errorHandler;
this.smiStream = new SitemapItemStream({ level: opts.level });
this.smiStream.on('data', (data) => this.push(data));
this.lastmodDateOnly = opts.lastmodDateOnly || false;
this.xmlNS = opts.xmlns || defaultXMLNS;
this.xslUrl = opts.xslUrl;
}
_transform(item, encoding, callback) {
if (!this.hasHeadOutput) {
this.hasHeadOutput = true;
this.push(getURLSetNs(this.xmlNS, this.xslUrl));
}
if (!this.smiStream.write(validateSMIOptions(normalizeURL(item, this.hostname, this.lastmodDateOnly), this.level, this.errorHandler))) {
this.smiStream.once('drain', callback);
}
else {
process.nextTick(callback);
}
}
_flush(cb) {
if (!this.hasHeadOutput) {
cb(new EmptySitemap());
}
else {
this.push(closetag);
cb();
}
}
}
/**
* Converts a readable stream into a promise that resolves with the concatenated data from the stream.
*
* The function listens for 'data' events from the stream, and when the stream ends, it resolves the promise with the concatenated data. If an error occurs while reading from the stream, the promise is rejected with the error.
*
* ⚠️ CAUTION: This function should not generally be used in production / when writing to files as it holds a copy of the entire file contents in memory until finished.
*
* @param {Readable} stream - The readable stream to convert to a promise.
* @returns {Promise<Buffer>} A promise that resolves with the concatenated data from the stream as a Buffer, or rejects with an error if one occurred while reading from the stream. If the stream is empty, the promise is rejected with an EmptyStream error.
* @throws {EmptyStream} If the stream is empty.
*/
export function streamToPromise(stream) {
return new Promise((resolve, reject) => {
const drain = [];
stream
// Error propagation is not automatic
// Bubble up errors on the read stream
.on('error', reject)
.pipe(new Writable({
write(chunk, enc, next) {
drain.push(chunk);
next();
},
}))
// This bubbles up errors when writing to the internal buffer
// This is unlikely to happen, but we have this for completeness
.on('error', reject)
.on('finish', () => {
if (!drain.length) {
reject(new EmptyStream());
}
else {
resolve(Buffer.concat(drain));
}
});
});
}
node_modules/sitemap/dist/esm/lib/sitemap-xml.d.ts
Generated Vendored
+107
View File
@@ -0,0 +1,107 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { TagNames, IndexTagNames, StringObj } from './types.js';
/**
* Escapes text content for safe inclusion in XML text nodes.
*
* **Security Model:**
* - Escapes `&` → `&amp;` (required to prevent entity interpretation)
* - Escapes `<` → `&lt;` (required to prevent tag injection)
* - Escapes `>` → `&gt;` (defense-in-depth, prevents CDATA injection)
* - Does NOT escape `"` or `'` (not required in text content, only in attributes)
* - Removes invalid XML Unicode characters per XML 1.0 spec
*
* **Why quotes aren't escaped:**
* In XML text content (between tags), quotes have no special meaning and don't
* need escaping. They only need escaping in attribute values, which is handled
* by the `otag()` function.
*
* @param txt - The text content to escape
* @returns XML-safe escaped text with invalid characters removed
* @throws {TypeError} If txt is not a string
*
* @example
* text('Hello & World'); // Returns: 'Hello &amp; World'
* text('5 < 10'); // Returns: '5 &lt; 10'
* text('Hello "World"'); // Returns: 'Hello "World"' (quotes OK in text)
*
* @see https://www.w3.org/TR/xml/#syntax
*/
export declare function text(txt: string): string;
/**
* Generates an opening XML tag with optional attributes.
*
* **Security Model:**
* - Validates attribute names to prevent injection via malformed names
* - Escapes all attribute values with proper XML entity encoding
* - Escapes `&`, `<`, `>`, `"`, and `'` in attribute values
* - Removes invalid XML Unicode characters
*
* Attribute values use full escaping (including quotes) because they appear
* within quoted strings in the XML output: `<tag attr="value">`.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @param attrs - Optional object mapping attribute names to string values
* @param selfClose - If true, generates a self-closing tag (e.g., `<tag/>`)
* @returns Opening XML tag string
* @throws {InvalidXMLAttributeNameError} If an attribute name contains invalid characters
* @throws {TypeError} If nodeName is not a string or attrs values are not strings
*
* @example
* otag('url'); // Returns: '<url>'
* otag('video:player_loc', { autoplay: 'ap=1' }); // Returns: '<video:player_loc autoplay="ap=1">'
* otag('image:image', {}, true); // Returns: '<image:image/>'
*
* @see https://www.w3.org/TR/xml/#NT-Attribute
*/
export declare function otag(nodeName: TagNames | IndexTagNames, attrs?: StringObj, selfClose?: boolean): string;
/**
* Generates a closing XML tag.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @returns Closing XML tag string
* @throws {TypeError} If nodeName is not a string
*
* @example
* ctag('url'); // Returns: '</url>'
* ctag('video:title'); // Returns: '</video:title>'
*/
export declare function ctag(nodeName: TagNames | IndexTagNames): string;
/**
* Generates a complete XML element with optional attributes and text content.
*
* This is a convenience function that combines `otag()`, `text()`, and `ctag()`.
* It supports three usage patterns via function overloading:
*
* 1. Element with text content: `element('loc', 'https://example.com')`
* 2. Element with attributes and text: `element('video:player_loc', { autoplay: 'ap=1' }, 'https://...')`
* 3. Self-closing element with attributes: `element('image:image', { href: '...' })`
*
* @param nodeName - The XML element name
* @param attrs - Either a string (text content) or object (attributes)
* @param innerText - Optional text content when attrs is an object
* @returns Complete XML element string
* @throws {InvalidXMLAttributeNameError} If an attribute name contains invalid characters
* @throws {TypeError} If arguments have invalid types
*
* @example
* // Pattern 1: Simple element with text
* element('loc', 'https://example.com')
* // Returns: '<loc>https://example.com</loc>'
*
* @example
* // Pattern 2: Element with attributes and text
* element('video:player_loc', { autoplay: 'ap=1' }, 'https://example.com/video')
* // Returns: '<video:player_loc autoplay="ap=1">https://example.com/video</video:player_loc>'
*
* @example
* // Pattern 3: Self-closing element with attributes
* element('xhtml:link', { rel: 'alternate', href: 'https://example.com/fr' })
* // Returns: '<xhtml:link rel="alternate" href="https://example.com/fr"/>'
*/
export declare function element(nodeName: TagNames, attrs: StringObj, innerText: string): string;
export declare function element(nodeName: TagNames | IndexTagNames, innerText: string): string;
export declare function element(nodeName: TagNames, attrs: StringObj): string;
node_modules/sitemap/dist/esm/lib/sitemap-xml.js
Generated Vendored
+181
View File
@@ -0,0 +1,181 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { InvalidXMLAttributeNameError } from './errors.js';
/**
* Regular expression matching invalid XML 1.0 Unicode characters that must be removed.
*
* Based on the XML 1.0 specification (https://www.w3.org/TR/xml/#charsets):
* - Control characters (U+0000-U+001F except tab, newline, carriage return)
* - Delete character (U+007F)
* - Invalid control characters (U+0080-U+009F except U+0085)
* - Surrogate pairs (U+D800-U+DFFF)
* - Non-characters (\p{NChar} - permanently reserved code points)
*
* Performance note: This regex uses Unicode property escapes and may be slower
* on very large strings (100KB+). Consider pre-validation for untrusted input.
*
* @see https://www.w3.org/TR/xml/#charsets
*/
const invalidXMLUnicodeRegex =
// eslint-disable-next-line no-control-regex
/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F-\u0084\u0086-\u009F\uD800-\uDFFF\p{NChar}]/gu;
/**
* Regular expressions for XML entity escaping
*/
const amp = /&/g;
const lt = /</g;
const gt = />/g;
const apos = /'/g;
const quot = /"/g;
/**
* Valid XML attribute name pattern. XML names must:
* - Start with a letter, underscore, or colon
* - Contain only letters, digits, hyphens, underscores, colons, or periods
*
* This is a simplified validation that accepts the most common attribute names.
* Note: In practice, this library only uses namespaced attributes like "video:title"
* which are guaranteed to be valid.
*
* @see https://www.w3.org/TR/xml/#NT-Name
*/
const validAttributeNameRegex = /^[a-zA-Z_:][\w:.-]*$/;
/**
* Validates that an attribute name is a valid XML identifier.
*
* XML attribute names must start with a letter, underscore, or colon,
* and contain only alphanumeric characters, hyphens, underscores, colons, or periods.
*
* @param name - The attribute name to validate
* @throws {InvalidXMLAttributeNameError} If the attribute name is invalid
*
* @example
* validateAttributeName('href'); // OK
* validateAttributeName('xml:lang'); // OK
* validateAttributeName('data-value'); // OK
* validateAttributeName('<script>'); // Throws InvalidXMLAttributeNameError
*/
function validateAttributeName(name) {
if (!validAttributeNameRegex.test(name)) {
throw new InvalidXMLAttributeNameError(name);
}
}
/**
* Escapes text content for safe inclusion in XML text nodes.
*
* **Security Model:**
* - Escapes `&` → `&amp;` (required to prevent entity interpretation)
* - Escapes `<` → `&lt;` (required to prevent tag injection)
* - Escapes `>` → `&gt;` (defense-in-depth, prevents CDATA injection)
* - Does NOT escape `"` or `'` (not required in text content, only in attributes)
* - Removes invalid XML Unicode characters per XML 1.0 spec
*
* **Why quotes aren't escaped:**
* In XML text content (between tags), quotes have no special meaning and don't
* need escaping. They only need escaping in attribute values, which is handled
* by the `otag()` function.
*
* @param txt - The text content to escape
* @returns XML-safe escaped text with invalid characters removed
* @throws {TypeError} If txt is not a string
*
* @example
* text('Hello & World'); // Returns: 'Hello &amp; World'
* text('5 < 10'); // Returns: '5 &lt; 10'
* text('Hello "World"'); // Returns: 'Hello "World"' (quotes OK in text)
*
* @see https://www.w3.org/TR/xml/#syntax
*/
export function text(txt) {
if (typeof txt !== 'string') {
throw new TypeError(`text() requires a string, received ${typeof txt}: ${String(txt)}`);
}
return txt
.replace(amp, '&amp;')
.replace(lt, '&lt;')
.replace(gt, '&gt;')
.replace(invalidXMLUnicodeRegex, '');
}
/**
* Generates an opening XML tag with optional attributes.
*
* **Security Model:**
* - Validates attribute names to prevent injection via malformed names
* - Escapes all attribute values with proper XML entity encoding
* - Escapes `&`, `<`, `>`, `"`, and `'` in attribute values
* - Removes invalid XML Unicode characters
*
* Attribute values use full escaping (including quotes) because they appear
* within quoted strings in the XML output: `<tag attr="value">`.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @param attrs - Optional object mapping attribute names to string values
* @param selfClose - If true, generates a self-closing tag (e.g., `<tag/>`)
* @returns Opening XML tag string
* @throws {InvalidXMLAttributeNameError} If an attribute name contains invalid characters
* @throws {TypeError} If nodeName is not a string or attrs values are not strings
*
* @example
* otag('url'); // Returns: '<url>'
* otag('video:player_loc', { autoplay: 'ap=1' }); // Returns: '<video:player_loc autoplay="ap=1">'
* otag('image:image', {}, true); // Returns: '<image:image/>'
*
* @see https://www.w3.org/TR/xml/#NT-Attribute
*/
export function otag(nodeName, attrs, selfClose = false) {
if (typeof nodeName !== 'string') {
throw new TypeError(`otag() nodeName must be a string, received ${typeof nodeName}: ${String(nodeName)}`);
}
let attrstr = '';
for (const k in attrs) {
// Validate attribute name to prevent injection
validateAttributeName(k);
const attrValue = attrs[k];
if (typeof attrValue !== 'string') {
throw new TypeError(`otag() attribute "${k}" value must be a string, received ${typeof attrValue}: ${String(attrValue)}`);
}
// Escape attribute value with full entity encoding
const val = attrValue
.replace(amp, '&amp;')
.replace(lt, '&lt;')
.replace(gt, '&gt;')
.replace(apos, '&apos;')
.replace(quot, '&quot;')
.replace(invalidXMLUnicodeRegex, '');
attrstr += ` ${k}="${val}"`;
}
return `<${nodeName}${attrstr}${selfClose ? '/' : ''}>`;
}
/**
* Generates a closing XML tag.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @returns Closing XML tag string
* @throws {TypeError} If nodeName is not a string
*
* @example
* ctag('url'); // Returns: '</url>'
* ctag('video:title'); // Returns: '</video:title>'
*/
export function ctag(nodeName) {
if (typeof nodeName !== 'string') {
throw new TypeError(`ctag() nodeName must be a string, received ${typeof nodeName}: ${String(nodeName)}`);
}
return `</${nodeName}>`;
}
export function element(nodeName, attrs, innerText) {
if (typeof attrs === 'string') {
// Pattern 1: element(nodeName, textContent)
return otag(nodeName) + text(attrs) + ctag(nodeName);
}
else if (innerText !== undefined) {
// Pattern 2: element(nodeName, attrs, textContent)
return otag(nodeName, attrs) + text(innerText) + ctag(nodeName);
}
else {
// Pattern 3: element(nodeName, attrs) - self-closing
return otag(nodeName, attrs, true);
}
}
node_modules/sitemap/dist/esm/lib/types.d.ts
Generated Vendored
+400
View File
@@ -0,0 +1,400 @@
import { URL } from 'node:url';
/**
* How frequently the page is likely to change. This value provides general
* information to search engines and may not correlate exactly to how often they crawl the page. Please note that the
* value of this tag is considered a hint and not a command. See
* <https://www.sitemaps.org/protocol.html#xmlTagDefinitions> for the acceptable
* values
*/
export declare enum EnumChangefreq {
DAILY = "daily",
MONTHLY = "monthly",
ALWAYS = "always",
HOURLY = "hourly",
WEEKLY = "weekly",
YEARLY = "yearly",
NEVER = "never"
}
export declare enum EnumYesNo {
YES = "YES",
NO = "NO",
Yes = "Yes",
No = "No",
yes = "yes",
no = "no"
}
export declare enum EnumAllowDeny {
ALLOW = "allow",
DENY = "deny"
}
/**
* https://support.google.com/webmasters/answer/74288?hl=en&ref_topic=4581190
*/
export interface NewsItem {
access?: 'Registration' | 'Subscription';
publication: {
name: string;
/**
* The `<language>` is the language of your publication. Use an ISO 639
* language code (2 or 3 letters).
*/
language: string;
};
/**
* @example 'PressRelease, Blog'
*/
genres?: string;
/**
* Article publication date in W3C format, using either the "complete date" (YYYY-MM-DD) format or the "complete date
* plus hours, minutes, and seconds"
*/
publication_date: string;
/**
* The title of the news article
* @example 'Companies A, B in Merger Talks'
*/
title: string;
/**
* @example 'business, merger, acquisition'
*/
keywords?: string;
/**
* @example 'NASDAQ:A, NASDAQ:B'
*/
stock_tickers?: string;
}
/**
* Sitemap Image
* https://support.google.com/webmasters/answer/178636?hl=en&ref_topic=4581190
*/
export interface Img {
/**
* The URL of the image
* @example 'https://example.com/image.jpg'
*/
url: string;
/**
* The caption of the image
* @example 'Thanksgiving dinner'
*/
caption?: string;
/**
* The title of the image
* @example 'Star Wars EP IV'
*/
title?: string;
/**
* The geographic location of the image.
* @example 'Limerick, Ireland'
*/
geoLocation?: string;
/**
* A URL to the license of the image.
* @example 'https://example.com/license.txt'
*/
license?: string;
}
interface VideoItemBase {
/**
* A URL pointing to the video thumbnail image file
* @example "https://rtv3-img-roosterteeth.akamaized.net/store/0e841100-289b-4184-ae30-b6a16736960a.jpg/sm/thumb3.jpg"
*/
thumbnail_loc: string;
/**
* The title of the video
* @example '2018:E6 - GoldenEye: Source'
*/
title: string;
/**
* A description of the video. Maximum 2048 characters.
* @example 'We play gun game in GoldenEye: Source with a good friend of ours. His name is Gruchy. Dan Gruchy.'
*/
description: string;
/**
* A URL pointing to the actual video media file. Should be one of the supported formats. HTML is not a supported
* format. Flash is allowed, but no longer supported on most mobile platforms, and so may be indexed less well. Must
* not be the same as the `<loc>` URL.
* @example "http://streamserver.example.com/video123.mp4"
*/
content_loc?: string;
/**
* A URL pointing to a player for a specific video. Usually this is the information in the src element of an `<embed>`
* tag. Must not be the same as the `<loc>` URL
* @example "https://roosterteeth.com/embed/rouletsplay-2018-goldeneye-source"
*/
player_loc?: string;
/**
* A string the search engine can append as a query param to enable automatic
* playback. Equivilant to auto play attr on player_loc tag.
* @example 'ap=1'
*/
'player_loc:autoplay'?: string;
/**
* Whether the search engine can embed the video in search results. Allowed values are yes or no.
*/
'player_loc:allow_embed'?: EnumYesNo;
/**
* The length of the video in seconds
* @example 600
*/
duration?: number;
/**
* The date after which the video will no longer be available.
* @example "2012-07-16T19:20:30+08:00"
*/
expiration_date?: string;
/**
* The number of times the video has been viewed
*/
view_count?: number;
/**
* The date the video was first published, in W3C format.
* @example "2012-07-16T19:20:30+08:00"
*/
publication_date?: string;
/**
* A short description of the broad category that the video belongs to. This is a string no longer than 256 characters.
* @example Baking
*/
category?: string;
/**
* Whether to show or hide your video in search results from specific countries.
* @example "IE GB US CA"
*/
restriction?: string;
/**
* Whether the countries in restriction are allowed or denied
* @example 'deny'
*/
'restriction:relationship'?: EnumAllowDeny;
gallery_loc?: string;
/**
* [Optional] Specifies the URL of a webpage with additional information about this uploader. This URL must be in the same domain as the <loc> tag.
* @see https://developers.google.com/search/docs/advanced/sitemaps/video-sitemaps
* @example http://www.example.com/users/grillymcgrillerson
*/
'uploader:info'?: string;
'gallery_loc:title'?: string;
/**
* The price to download or view the video. Omit this tag for free videos.
* @example "1.99"
*/
price?: string;
/**
* Specifies the resolution of the purchased version. Supported values are hd and sd.
* @example "HD"
*/
'price:resolution'?: Resolution;
/**
* Specifies the currency in ISO4217 format.
* @example "USD"
*/
'price:currency'?: string;
/**
* Specifies the purchase option. Supported values are rend and own.
* @example "rent"
*/
'price:type'?: PriceType;
/**
* The video uploader's name. Only one <video:uploader> is allowed per video. String value, max 255 characters.
* @example "GrillyMcGrillerson"
*/
uploader?: string;
/**
* Whether to show or hide your video in search results on specified platform types. This is a list of space-delimited
* platform types. See <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190> for more detail
* @example "tv"
*/
platform?: string;
id?: string;
'platform:relationship'?: EnumAllowDeny;
}
/**
* Video price type - supports both lowercase and uppercase variants
* as allowed by the Google Video Sitemap specification
* @see https://developers.google.com/search/docs/advanced/sitemaps/video-sitemaps
*/
export type PriceType = 'rent' | 'purchase' | 'RENT' | 'PURCHASE';
/**
* Video resolution - supports both lowercase and uppercase variants
* as allowed by the Google Video Sitemap specification
* @see https://developers.google.com/search/docs/advanced/sitemaps/video-sitemaps
*/
export type Resolution = 'HD' | 'hd' | 'sd' | 'SD';
/**
* Sitemap video. <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190>
*/
export interface VideoItem extends VideoItemBase {
/**
* An arbitrary string tag describing the video. Tags are generally very short descriptions of key concepts associated
* with a video or piece of content.
* @example ['Baking']
*/
tag: string[];
/**
* The rating of the video. Supported values are float numbers.
* @example 2.5
*/
rating?: number;
family_friendly?: EnumYesNo;
/**
* Indicates whether a subscription (either paid or free) is required to view
* the video. Allowed values are yes or no.
*/
requires_subscription?: EnumYesNo;
/**
* Indicates whether the video is a live stream. Supported values are yes or no.
*/
live?: EnumYesNo;
}
/**
* Sitemap video. <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190>
*/
export interface VideoItemLoose extends VideoItemBase {
/**
* An arbitrary string tag describing the video. Tags are generally very short descriptions of key concepts associated
* with a video or piece of content.
* @example ['Baking']
*/
tag?: string | string[];
/**
* The rating of the video. Supported values are float numbers.
* @example 2.5
*/
rating?: string | number;
family_friendly?: EnumYesNo | boolean;
requires_subscription?: EnumYesNo | boolean;
/**
* Indicates whether the video is a live stream. Supported values are yes or no.
*/
live?: EnumYesNo | boolean;
}
/**
* https://support.google.com/webmasters/answer/189077
*/
export interface LinkItem {
/**
* @example 'en'
*/
lang: string;
/**
* @example 'en-us'
*/
hreflang?: string;
url: string;
}
export interface IndexItem {
url: string;
lastmod?: string;
}
interface SitemapItemBase {
lastmod?: string;
changefreq?: EnumChangefreq;
fullPrecisionPriority?: boolean;
priority?: number;
news?: NewsItem;
expires?: string;
androidLink?: string;
ampLink?: string;
url: string;
}
/**
* Strict options for individual sitemap entries
*/
export interface SitemapItem extends SitemapItemBase {
img: Img[];
video: VideoItem[];
links: LinkItem[];
}
/**
* Options for individual sitemap entries prior to normalization
*/
export interface SitemapItemLoose extends SitemapItemBase {
video?: VideoItemLoose | VideoItemLoose[];
img?: string | Img | (string | Img)[];
links?: LinkItem[];
lastmodfile?: string | Buffer | URL;
lastmodISO?: string;
lastmodrealtime?: boolean;
}
/**
* How to handle errors in passed in urls
*/
export declare enum ErrorLevel {
/**
* Validation will be skipped and nothing logged or thrown.
*/
SILENT = "silent",
/**
* If an invalid value is encountered, a console.warn will be called with details
*/
WARN = "warn",
/**
* An Error will be thrown on encountering invalid data.
*/
THROW = "throw"
}
export type ErrorHandler = (error: Error, level: ErrorLevel) => void;
export declare enum TagNames {
url = "url",
loc = "loc",
urlset = "urlset",
lastmod = "lastmod",
changefreq = "changefreq",
priority = "priority",
'video:thumbnail_loc' = "video:thumbnail_loc",
'video:video' = "video:video",
'video:title' = "video:title",
'video:description' = "video:description",
'video:tag' = "video:tag",
'video:duration' = "video:duration",
'video:player_loc' = "video:player_loc",
'video:content_loc' = "video:content_loc",
'image:image' = "image:image",
'image:loc' = "image:loc",
'image:geo_location' = "image:geo_location",
'image:license' = "image:license",
'image:title' = "image:title",
'image:caption' = "image:caption",
'video:requires_subscription' = "video:requires_subscription",
'video:publication_date' = "video:publication_date",
'video:id' = "video:id",
'video:restriction' = "video:restriction",
'video:family_friendly' = "video:family_friendly",
'video:view_count' = "video:view_count",
'video:uploader' = "video:uploader",
'video:expiration_date' = "video:expiration_date",
'video:platform' = "video:platform",
'video:price' = "video:price",
'video:rating' = "video:rating",
'video:category' = "video:category",
'video:live' = "video:live",
'video:gallery_loc' = "video:gallery_loc",
'news:news' = "news:news",
'news:publication' = "news:publication",
'news:name' = "news:name",
'news:access' = "news:access",
'news:genres' = "news:genres",
'news:publication_date' = "news:publication_date",
'news:title' = "news:title",
'news:keywords' = "news:keywords",
'news:stock_tickers' = "news:stock_tickers",
'news:language' = "news:language",
'mobile:mobile' = "mobile:mobile",
'xhtml:link' = "xhtml:link",
'expires' = "expires"
}
export declare enum IndexTagNames {
sitemap = "sitemap",
sitemapindex = "sitemapindex",
loc = "loc",
lastmod = "lastmod"
}
/**
* Generic object with string keys and any values
* Used for XML attribute building and other flexible data structures
*/
export interface StringObj {
[index: string]: any;
}
export {};
node_modules/sitemap/dist/esm/lib/types.js
Generated Vendored
+106
View File
@@ -0,0 +1,106 @@
/**
* How frequently the page is likely to change. This value provides general
* information to search engines and may not correlate exactly to how often they crawl the page. Please note that the
* value of this tag is considered a hint and not a command. See
* <https://www.sitemaps.org/protocol.html#xmlTagDefinitions> for the acceptable
* values
*/
export var EnumChangefreq;
(function (EnumChangefreq) {
EnumChangefreq["DAILY"] = "daily";
EnumChangefreq["MONTHLY"] = "monthly";
EnumChangefreq["ALWAYS"] = "always";
EnumChangefreq["HOURLY"] = "hourly";
EnumChangefreq["WEEKLY"] = "weekly";
EnumChangefreq["YEARLY"] = "yearly";
EnumChangefreq["NEVER"] = "never";
})(EnumChangefreq || (EnumChangefreq = {}));
export var EnumYesNo;
(function (EnumYesNo) {
EnumYesNo["YES"] = "YES";
EnumYesNo["NO"] = "NO";
EnumYesNo["Yes"] = "Yes";
EnumYesNo["No"] = "No";
EnumYesNo["yes"] = "yes";
EnumYesNo["no"] = "no";
})(EnumYesNo || (EnumYesNo = {}));
export var EnumAllowDeny;
(function (EnumAllowDeny) {
EnumAllowDeny["ALLOW"] = "allow";
EnumAllowDeny["DENY"] = "deny";
})(EnumAllowDeny || (EnumAllowDeny = {}));
/**
* How to handle errors in passed in urls
*/
export var ErrorLevel;
(function (ErrorLevel) {
/**
* Validation will be skipped and nothing logged or thrown.
*/
ErrorLevel["SILENT"] = "silent";
/**
* If an invalid value is encountered, a console.warn will be called with details
*/
ErrorLevel["WARN"] = "warn";
/**
* An Error will be thrown on encountering invalid data.
*/
ErrorLevel["THROW"] = "throw";
})(ErrorLevel || (ErrorLevel = {}));
export var TagNames;
(function (TagNames) {
TagNames["url"] = "url";
TagNames["loc"] = "loc";
TagNames["urlset"] = "urlset";
TagNames["lastmod"] = "lastmod";
TagNames["changefreq"] = "changefreq";
TagNames["priority"] = "priority";
TagNames["video:thumbnail_loc"] = "video:thumbnail_loc";
TagNames["video:video"] = "video:video";
TagNames["video:title"] = "video:title";
TagNames["video:description"] = "video:description";
TagNames["video:tag"] = "video:tag";
TagNames["video:duration"] = "video:duration";
TagNames["video:player_loc"] = "video:player_loc";
TagNames["video:content_loc"] = "video:content_loc";
TagNames["image:image"] = "image:image";
TagNames["image:loc"] = "image:loc";
TagNames["image:geo_location"] = "image:geo_location";
TagNames["image:license"] = "image:license";
TagNames["image:title"] = "image:title";
TagNames["image:caption"] = "image:caption";
TagNames["video:requires_subscription"] = "video:requires_subscription";
TagNames["video:publication_date"] = "video:publication_date";
TagNames["video:id"] = "video:id";
TagNames["video:restriction"] = "video:restriction";
TagNames["video:family_friendly"] = "video:family_friendly";
TagNames["video:view_count"] = "video:view_count";
TagNames["video:uploader"] = "video:uploader";
TagNames["video:expiration_date"] = "video:expiration_date";
TagNames["video:platform"] = "video:platform";
TagNames["video:price"] = "video:price";
TagNames["video:rating"] = "video:rating";
TagNames["video:category"] = "video:category";
TagNames["video:live"] = "video:live";
TagNames["video:gallery_loc"] = "video:gallery_loc";
TagNames["news:news"] = "news:news";
TagNames["news:publication"] = "news:publication";
TagNames["news:name"] = "news:name";
TagNames["news:access"] = "news:access";
TagNames["news:genres"] = "news:genres";
TagNames["news:publication_date"] = "news:publication_date";
TagNames["news:title"] = "news:title";
TagNames["news:keywords"] = "news:keywords";
TagNames["news:stock_tickers"] = "news:stock_tickers";
TagNames["news:language"] = "news:language";
TagNames["mobile:mobile"] = "mobile:mobile";
TagNames["xhtml:link"] = "xhtml:link";
TagNames["expires"] = "expires";
})(TagNames || (TagNames = {}));
export var IndexTagNames;
(function (IndexTagNames) {
IndexTagNames["sitemap"] = "sitemap";
IndexTagNames["sitemapindex"] = "sitemapindex";
IndexTagNames["loc"] = "loc";
IndexTagNames["lastmod"] = "lastmod";
})(IndexTagNames || (IndexTagNames = {}));
node_modules/sitemap/dist/esm/lib/utils.d.ts
Generated Vendored
+48
View File
@@ -0,0 +1,48 @@
import { Readable, ReadableOptions, TransformOptions } from 'node:stream';
import { SitemapItem, SitemapItemLoose } from './types.js';
export { validateSMIOptions } from './validation.js';
/**
* Combines multiple streams into one
* @param streams the streams to combine
*/
export declare function mergeStreams(streams: Readable[], options?: TransformOptions): Readable;
export interface ReadlineStreamOptions extends ReadableOptions {
input: Readable;
}
/**
* Wraps node's ReadLine in a stream
*/
export declare class ReadlineStream extends Readable {
private _source;
constructor(options: ReadlineStreamOptions);
_read(size: number): void;
}
/**
* Takes a stream likely from fs.createReadStream('./path') and returns a stream
* of sitemap items
* @param stream a stream of line separated urls.
* @param opts.isJSON is the stream line separated JSON. leave undefined to guess
*/
export declare function lineSeparatedURLsToSitemapOptions(stream: Readable, { isJSON }?: {
isJSON?: boolean;
}): Readable;
/**
* Based on lodash's implementation of chunk.
*
* Copyright JS Foundation and other contributors <https://js.foundation/>
*
* Based on Underscore.js, copyright Jeremy Ashkenas,
* DocumentCloud and Investigative Reporters & Editors <http://underscorejs.org/>
*
* This software consists of voluntary contributions made by many
* individuals. For exact contribution history, see the revision history
* available at https://github.com/lodash/lodash
*/
export declare function chunk(array: any[], size?: number): any[];
/**
* Converts the passed in sitemap entry into one capable of being consumed by SitemapItem
* @param {string | SitemapItemLoose} elem the string or object to be converted
* @param {string} hostname
* @returns SitemapItemOptions a strict sitemap item option
*/
export declare function normalizeURL(elem: string | SitemapItemLoose, hostname?: string, lastmodDateOnly?: boolean): SitemapItem;
node_modules/sitemap/dist/esm/lib/utils.js
Generated Vendored
+221
View File
@@ -0,0 +1,221 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { statSync } from 'node:fs';
import { Readable, Transform, PassThrough, } from 'node:stream';
import { createInterface } from 'node:readline';
import { URL } from 'node:url';
import { EnumYesNo, } from './types.js';
// Re-export validateSMIOptions from validation.ts for backward compatibility
export { validateSMIOptions } from './validation.js';
/**
* Combines multiple streams into one
* @param streams the streams to combine
*/
export function mergeStreams(streams, options) {
let pass = new PassThrough(options);
let waiting = streams.length;
for (const stream of streams) {
pass = stream.pipe(pass, { end: false });
stream.once('end', () => --waiting === 0 && pass.emit('end'));
}
return pass;
}
/**
* Wraps node's ReadLine in a stream
*/
export class ReadlineStream extends Readable {
_source;
constructor(options) {
if (options.autoDestroy === undefined) {
options.autoDestroy = true;
}
options.objectMode = true;
super(options);
this._source = createInterface({
input: options.input,
terminal: false,
crlfDelay: Infinity,
});
// Every time there's data, push it into the internal buffer.
this._source.on('line', (chunk) => {
// If push() returns false, then stop reading from source.
if (!this.push(chunk))
this._source.pause();
});
// When the source ends, push the EOF-signaling `null` chunk.
this._source.on('close', () => {
this.push(null);
});
}
// _read() will be called when the stream wants to pull more data in.
// The advisory size argument is ignored in this case.
_read(size) {
this._source.resume();
}
}
/**
* Takes a stream likely from fs.createReadStream('./path') and returns a stream
* of sitemap items
* @param stream a stream of line separated urls.
* @param opts.isJSON is the stream line separated JSON. leave undefined to guess
*/
export function lineSeparatedURLsToSitemapOptions(stream, { isJSON } = {}) {
return new ReadlineStream({ input: stream }).pipe(new Transform({
objectMode: true,
transform: (line, encoding, cb) => {
if (isJSON || (isJSON === undefined && line[0] === '{')) {
cb(null, JSON.parse(line));
}
else {
cb(null, line);
}
},
}));
}
/**
* Based on lodash's implementation of chunk.
*
* Copyright JS Foundation and other contributors <https://js.foundation/>
*
* Based on Underscore.js, copyright Jeremy Ashkenas,
* DocumentCloud and Investigative Reporters & Editors <http://underscorejs.org/>
*
* This software consists of voluntary contributions made by many
* individuals. For exact contribution history, see the revision history
* available at https://github.com/lodash/lodash
*/
/* eslint-disable @typescript-eslint/no-explicit-any */
export function chunk(array, size = 1) {
size = Math.max(Math.trunc(size), 0);
const length = array ? array.length : 0;
if (!length || size < 1) {
return [];
}
const result = Array(Math.ceil(length / size));
let index = 0, resIndex = 0;
while (index < length) {
result[resIndex++] = array.slice(index, (index += size));
}
return result;
}
function boolToYESNO(bool) {
if (bool === undefined) {
return undefined;
}
if (typeof bool === 'boolean') {
return bool ? EnumYesNo.yes : EnumYesNo.no;
}
return bool;
}
/**
* Converts the passed in sitemap entry into one capable of being consumed by SitemapItem
* @param {string | SitemapItemLoose} elem the string or object to be converted
* @param {string} hostname
* @returns SitemapItemOptions a strict sitemap item option
*/
export function normalizeURL(elem, hostname, lastmodDateOnly = false) {
// SitemapItem
// create object with url property
const smi = {
img: [],
video: [],
links: [],
url: '',
};
if (typeof elem === 'string') {
smi.url = new URL(elem, hostname).toString();
return smi;
}
const { url, img, links, video, lastmodfile, lastmodISO, lastmod, ...other } = elem;
Object.assign(smi, other);
smi.url = new URL(url, hostname).toString();
if (img) {
// prepend hostname to all image urls
smi.img = (Array.isArray(img) ? img : [img]).map((el) => typeof el === 'string'
? { url: new URL(el, hostname).toString() }
: { ...el, url: new URL(el.url, hostname).toString() });
}
if (links) {
smi.links = links.map((link) => ({
...link,
url: new URL(link.url, hostname).toString(),
}));
}
if (video) {
smi.video = (Array.isArray(video) ? video : [video]).map((video) => {
const nv = {
...video,
family_friendly: boolToYESNO(video.family_friendly),
live: boolToYESNO(video.live),
requires_subscription: boolToYESNO(video.requires_subscription),
tag: [],
rating: undefined,
};
if (video.tag !== undefined) {
nv.tag = !Array.isArray(video.tag) ? [video.tag] : video.tag;
}
if (video.rating !== undefined) {
if (typeof video.rating === 'string') {
const parsedRating = parseFloat(video.rating);
// Validate parsed rating is a valid number
if (Number.isNaN(parsedRating)) {
throw new Error(`Invalid video rating "${video.rating}" for URL "${elem.url}": must be a valid number`);
}
nv.rating = parsedRating;
}
else {
nv.rating = video.rating;
}
}
if (typeof video.view_count === 'string') {
const parsedViewCount = parseInt(video.view_count, 10);
// Validate parsed view count is a valid non-negative integer
if (Number.isNaN(parsedViewCount)) {
throw new Error(`Invalid video view_count "${video.view_count}" for URL "${elem.url}": must be a valid number`);
}
if (parsedViewCount < 0) {
throw new Error(`Invalid video view_count "${video.view_count}" for URL "${elem.url}": cannot be negative`);
}
nv.view_count = parsedViewCount;
}
else if (typeof video.view_count === 'number') {
nv.view_count = video.view_count;
}
return nv;
});
}
// If given a file to use for last modified date
if (lastmodfile) {
const { mtime } = statSync(lastmodfile);
const lastmodDate = new Date(mtime);
// Validate date is valid
if (Number.isNaN(lastmodDate.getTime())) {
throw new Error(`Invalid date from file stats for URL "${smi.url}": file modification time is invalid`);
}
smi.lastmod = lastmodDate.toISOString();
// The date of last modification (YYYY-MM-DD)
}
else if (lastmodISO) {
const lastmodDate = new Date(lastmodISO);
// Validate date is valid
if (Number.isNaN(lastmodDate.getTime())) {
throw new Error(`Invalid lastmodISO "${lastmodISO}" for URL "${smi.url}": must be a valid date string`);
}
smi.lastmod = lastmodDate.toISOString();
}
else if (lastmod) {
const lastmodDate = new Date(lastmod);
// Validate date is valid
if (Number.isNaN(lastmodDate.getTime())) {
throw new Error(`Invalid lastmod "${lastmod}" for URL "${smi.url}": must be a valid date string`);
}
smi.lastmod = lastmodDate.toISOString();
}
if (lastmodDateOnly && smi.lastmod) {
smi.lastmod = smi.lastmod.slice(0, 10);
}
return smi;
}
node_modules/sitemap/dist/esm/lib/validation.d.ts
Generated Vendored
+94
View File
@@ -0,0 +1,94 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { SitemapItem, ErrorLevel, EnumChangefreq, EnumYesNo, EnumAllowDeny, PriceType, Resolution, ErrorHandler } from './types.js';
export declare const validators: {
[index: string]: RegExp;
};
/**
* Type guard to check if a string is a valid price type
*/
export declare function isPriceType(pt: string | PriceType): pt is PriceType;
/**
* Type guard to check if a string is a valid resolution
*/
export declare function isResolution(res: string): res is Resolution;
export declare function isValidChangeFreq(freq: string): freq is EnumChangefreq;
/**
* Type guard to check if a string is a valid yes/no value
*/
export declare function isValidYesNo(yn: string): yn is EnumYesNo;
/**
* Type guard to check if a string is a valid allow/deny value
*/
export declare function isAllowDeny(ad: string): ad is EnumAllowDeny;
/**
* Validates that a URL is well-formed and meets security requirements
*
* Security: This function enforces that URLs use safe protocols (http/https),
* are within reasonable length limits (2048 chars per sitemaps.org spec),
* and can be properly parsed. This prevents protocol injection attacks and
* ensures compliance with sitemap specifications.
*
* @param url - The URL to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidHostnameError} If the URL is invalid
*/
export declare function validateURL(url: string, paramName: string): void;
/**
* Validates that a path doesn't contain path traversal sequences
*
* Security: This function prevents path traversal attacks by detecting
* any occurrence of '..' in the path, whether it appears as '../', '/..',
* or standalone. This prevents attackers from accessing files outside
* the intended directory structure.
*
* @param path - The path to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidPathError} If the path contains traversal sequences
*/
export declare function validatePath(path: string, paramName: string): void;
/**
* Validates that a public base path is safe for URL construction
*
* Security: This function prevents path traversal attacks and validates
* that the path is safe for use in URL construction within sitemap indexes.
* It checks for '..' sequences, null bytes, and invalid whitespace that
* could be used to manipulate URL structure or inject malicious content.
*
* @param publicBasePath - The public base path to validate
* @throws {InvalidPublicBasePathError} If the path is invalid
*/
export declare function validatePublicBasePath(publicBasePath: string): void;
/**
* Validates that a limit is within acceptable range per sitemaps.org spec
*
* Security: This function enforces sitemap size limits (1-50,000 URLs per
* sitemap) as specified by sitemaps.org. This prevents resource exhaustion
* attacks and ensures compliance with search engine requirements.
*
* @param limit - The limit to validate
* @throws {InvalidLimitError} If the limit is out of range
*/
export declare function validateLimit(limit: number): void;
/**
* Validates that an XSL URL is safe and well-formed
*
* Security: This function validates XSL stylesheet URLs to prevent
* injection attacks. It blocks dangerous protocols and content patterns
* that could be used for XSS or other attacks. The validation uses
* case-insensitive matching to catch obfuscated attacks.
*
* @param xslUrl - The XSL URL to validate
* @throws {InvalidXSLUrlError} If the URL is invalid
*/
export declare function validateXSLUrl(xslUrl: string): void;
/**
* Verifies all data passed in will comply with sitemap spec.
* @param conf Options to validate
* @param level logging level
* @param errorHandler error handling func
*/
export declare function validateSMIOptions(conf: SitemapItem, level?: ErrorLevel, errorHandler?: ErrorHandler): SitemapItem;
node_modules/sitemap/dist/esm/lib/validation.js
Generated Vendored
+384
View File
@@ -0,0 +1,384 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { InvalidPathError, InvalidHostnameError, InvalidLimitError, InvalidPublicBasePathError, InvalidXSLUrlError, ChangeFreqInvalidError, InvalidAttrValue, InvalidNewsAccessValue, InvalidNewsFormat, InvalidVideoDescription, InvalidVideoDuration, InvalidVideoFormat, InvalidVideoRating, NoURLError, NoConfigError, PriorityInvalidError, InvalidVideoTitle, InvalidVideoViewCount, InvalidVideoTagCount, InvalidVideoCategory, InvalidVideoFamilyFriendly, InvalidVideoRestriction, InvalidVideoRestrictionRelationship, InvalidVideoPriceType, InvalidVideoResolution, InvalidVideoPriceCurrency, } from './errors.js';
import { ErrorLevel, EnumChangefreq, } from './types.js';
import { LIMITS } from './constants.js';
import { isAbsolute } from 'node:path';
/**
* Validator regular expressions for various sitemap fields
*/
const allowDeny = /^(?:allow|deny)$/;
export const validators = {
'price:currency': /^[A-Z]{3}$/,
'price:type': /^(?:rent|purchase|RENT|PURCHASE)$/,
'price:resolution': /^(?:HD|hd|sd|SD)$/,
'platform:relationship': allowDeny,
'restriction:relationship': allowDeny,
restriction: /^([A-Z]{2}( +[A-Z]{2})*)?$/,
platform: /^((web|mobile|tv)( (web|mobile|tv))*)?$/,
// Language codes: zh-cn, zh-tw, or ISO 639 2-3 letter codes
language: /^(zh-cn|zh-tw|[a-z]{2,3})$/,
genres: /^(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated)(, *(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated))*$/,
stock_tickers: /^(\w+:\w+(, *\w+:\w+){0,4})?$/,
};
/**
* Type guard to check if a string is a valid price type
*/
export function isPriceType(pt) {
return validators['price:type'].test(pt);
}
/**
* Type guard to check if a string is a valid resolution
*/
export function isResolution(res) {
return validators['price:resolution'].test(res);
}
/**
* Type guard to check if a string is a valid changefreq value
*/
const CHANGEFREQ = Object.values(EnumChangefreq);
export function isValidChangeFreq(freq) {
return CHANGEFREQ.includes(freq);
}
/**
* Type guard to check if a string is a valid yes/no value
*/
export function isValidYesNo(yn) {
return /^YES|NO|[Yy]es|[Nn]o$/.test(yn);
}
/**
* Type guard to check if a string is a valid allow/deny value
*/
export function isAllowDeny(ad) {
return allowDeny.test(ad);
}
/**
* Validates that a URL is well-formed and meets security requirements
*
* Security: This function enforces that URLs use safe protocols (http/https),
* are within reasonable length limits (2048 chars per sitemaps.org spec),
* and can be properly parsed. This prevents protocol injection attacks and
* ensures compliance with sitemap specifications.
*
* @param url - The URL to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidHostnameError} If the URL is invalid
*/
export function validateURL(url, paramName) {
if (!url || typeof url !== 'string') {
throw new InvalidHostnameError(url, `${paramName} must be a non-empty string`);
}
if (url.length > LIMITS.MAX_URL_LENGTH) {
throw new InvalidHostnameError(url, `${paramName} exceeds maximum length of ${LIMITS.MAX_URL_LENGTH} characters`);
}
if (!LIMITS.URL_PROTOCOL_REGEX.test(url)) {
throw new InvalidHostnameError(url, `${paramName} must use http:// or https:// protocol`);
}
// Validate URL can be parsed
try {
new URL(url);
}
catch (err) {
throw new InvalidHostnameError(url, `${paramName} is not a valid URL: ${err instanceof Error ? err.message : String(err)}`);
}
}
/**
* Validates that a path doesn't contain path traversal sequences
*
* Security: This function prevents path traversal attacks by detecting
* any occurrence of '..' in the path, whether it appears as '../', '/..',
* or standalone. This prevents attackers from accessing files outside
* the intended directory structure.
*
* @param path - The path to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidPathError} If the path contains traversal sequences
*/
export function validatePath(path, paramName) {
if (!path || typeof path !== 'string') {
throw new InvalidPathError(path, `${paramName} must be a non-empty string`);
}
// Reject absolute paths to prevent arbitrary write location when caller input
// reaches destinationDir (BB-04)
if (isAbsolute(path)) {
throw new InvalidPathError(path, `${paramName} must be a relative path (absolute paths are not allowed)`);
}
// Check for path traversal sequences - must check before and after normalization
// to catch both Windows-style (\) and Unix-style (/) separators
if (path.includes('..')) {
throw new InvalidPathError(path, `${paramName} contains path traversal sequence (..)`);
}
// Additional check after normalization to catch encoded or obfuscated attempts
const normalizedPath = path.replace(/\\/g, '/');
const pathComponents = normalizedPath.split('/').filter((p) => p.length > 0);
if (pathComponents.includes('..')) {
throw new InvalidPathError(path, `${paramName} contains path traversal sequence (..)`);
}
// Check for null bytes (security issue in some contexts)
if (path.includes('\0')) {
throw new InvalidPathError(path, `${paramName} contains null byte character`);
}
}
/**
* Validates that a public base path is safe for URL construction
*
* Security: This function prevents path traversal attacks and validates
* that the path is safe for use in URL construction within sitemap indexes.
* It checks for '..' sequences, null bytes, and invalid whitespace that
* could be used to manipulate URL structure or inject malicious content.
*
* @param publicBasePath - The public base path to validate
* @throws {InvalidPublicBasePathError} If the path is invalid
*/
export function validatePublicBasePath(publicBasePath) {
if (!publicBasePath || typeof publicBasePath !== 'string') {
throw new InvalidPublicBasePathError(publicBasePath, 'must be a non-empty string');
}
// Check for path traversal - check the raw string first
if (publicBasePath.includes('..')) {
throw new InvalidPublicBasePathError(publicBasePath, 'contains path traversal sequence (..)');
}
// Additional check for path components after normalization
const normalizedPath = publicBasePath.replace(/\\/g, '/');
const pathComponents = normalizedPath.split('/').filter((p) => p.length > 0);
if (pathComponents.includes('..')) {
throw new InvalidPublicBasePathError(publicBasePath, 'contains path traversal sequence (..)');
}
// Check for null bytes
if (publicBasePath.includes('\0')) {
throw new InvalidPublicBasePathError(publicBasePath, 'contains null byte character');
}
// Check for potentially dangerous characters that could break URL construction
if (/[\r\n\t]/.test(publicBasePath)) {
throw new InvalidPublicBasePathError(publicBasePath, 'contains invalid whitespace characters');
}
}
/**
* Validates that a limit is within acceptable range per sitemaps.org spec
*
* Security: This function enforces sitemap size limits (1-50,000 URLs per
* sitemap) as specified by sitemaps.org. This prevents resource exhaustion
* attacks and ensures compliance with search engine requirements.
*
* @param limit - The limit to validate
* @throws {InvalidLimitError} If the limit is out of range
*/
export function validateLimit(limit) {
if (typeof limit !== 'number' ||
!Number.isFinite(limit) ||
Number.isNaN(limit)) {
throw new InvalidLimitError(limit);
}
if (limit < LIMITS.MIN_SITEMAP_ITEM_LIMIT ||
limit > LIMITS.MAX_SITEMAP_ITEM_LIMIT) {
throw new InvalidLimitError(limit);
}
// Ensure it's an integer
if (!Number.isInteger(limit)) {
throw new InvalidLimitError(limit);
}
}
/**
* Validates that an XSL URL is safe and well-formed
*
* Security: This function validates XSL stylesheet URLs to prevent
* injection attacks. It blocks dangerous protocols and content patterns
* that could be used for XSS or other attacks. The validation uses
* case-insensitive matching to catch obfuscated attacks.
*
* @param xslUrl - The XSL URL to validate
* @throws {InvalidXSLUrlError} If the URL is invalid
*/
export function validateXSLUrl(xslUrl) {
if (!xslUrl || typeof xslUrl !== 'string') {
throw new InvalidXSLUrlError(xslUrl, 'must be a non-empty string');
}
if (xslUrl.length > LIMITS.MAX_URL_LENGTH) {
throw new InvalidXSLUrlError(xslUrl, `exceeds maximum length of ${LIMITS.MAX_URL_LENGTH} characters`);
}
if (!LIMITS.URL_PROTOCOL_REGEX.test(xslUrl)) {
throw new InvalidXSLUrlError(xslUrl, 'must use http:// or https:// protocol');
}
// Validate URL can be parsed
try {
new URL(xslUrl);
}
catch (err) {
throw new InvalidXSLUrlError(xslUrl, `is not a valid URL: ${err instanceof Error ? err.message : String(err)}`);
}
// Check for potentially dangerous content (case-insensitive)
const lowerUrl = xslUrl.toLowerCase();
// Block dangerous HTML/script content
if (lowerUrl.includes('<script')) {
throw new InvalidXSLUrlError(xslUrl, 'contains potentially malicious content (<script tag)');
}
// Block dangerous protocols (already checked http/https above, but double-check for encoded variants)
const dangerousProtocols = [
'javascript:',
'data:',
'vbscript:',
'file:',
'about:',
];
for (const protocol of dangerousProtocols) {
if (lowerUrl.includes(protocol)) {
throw new InvalidXSLUrlError(xslUrl, `contains dangerous protocol: ${protocol}`);
}
}
// Check for URL-encoded variants of dangerous patterns
// %3C = '<', %3E = '>', %3A = ':'
const encodedPatterns = [
'%3cscript', // <script
'%3c%73%63%72%69%70%74', // <script (fully encoded)
'javascript%3a', // javascript:
'data%3a', // data:
];
for (const pattern of encodedPatterns) {
if (lowerUrl.includes(pattern)) {
throw new InvalidXSLUrlError(xslUrl, 'contains URL-encoded malicious content');
}
}
// Reject unencoded XML special characters — these must be percent-encoded in
// valid URLs and could break out of XML attribute context if left raw.
if (xslUrl.includes('"') || xslUrl.includes('<') || xslUrl.includes('>')) {
throw new InvalidXSLUrlError(xslUrl, 'contains unencoded XML special characters (" < >); percent-encode them in the URL');
}
}
/**
* Internal helper to validate fields against their validators
*/
function validate(subject, name, url, level) {
Object.keys(subject).forEach((key) => {
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
const val = subject[key];
if (validators[key] && !validators[key].test(val)) {
if (level === ErrorLevel.THROW) {
throw new InvalidAttrValue(key, val, validators[key]);
}
else {
console.warn(`${url}: ${name} key ${key} has invalid value: ${val}`);
}
}
});
}
/**
* Internal helper to handle errors based on error level
*/
function handleError(error, level) {
if (level === ErrorLevel.THROW) {
throw error;
}
else if (level === ErrorLevel.WARN) {
console.warn(error.name, error.message);
}
}
/**
* Verifies all data passed in will comply with sitemap spec.
* @param conf Options to validate
* @param level logging level
* @param errorHandler error handling func
*/
export function validateSMIOptions(conf, level = ErrorLevel.WARN, errorHandler = handleError) {
if (!conf) {
throw new NoConfigError();
}
if (level === ErrorLevel.SILENT) {
return conf;
}
const { url, changefreq, priority, news, video } = conf;
if (!url) {
errorHandler(new NoURLError(), level);
}
if (changefreq) {
if (!isValidChangeFreq(changefreq)) {
errorHandler(new ChangeFreqInvalidError(url, changefreq), level);
}
}
if (priority) {
if (!(priority >= 0.0 && priority <= 1.0)) {
errorHandler(new PriorityInvalidError(url, priority), level);
}
}
if (news) {
if (news.access &&
news.access !== 'Registration' &&
news.access !== 'Subscription') {
errorHandler(new InvalidNewsAccessValue(url, news.access), level);
}
if (!news.publication ||
!news.publication.name ||
!news.publication.language ||
!news.publication_date ||
!news.title) {
errorHandler(new InvalidNewsFormat(url), level);
}
validate(news, 'news', url, level);
validate(news.publication, 'publication', url, level);
}
if (video) {
video.forEach((vid) => {
if (vid.duration !== undefined) {
if (vid.duration < 0 || vid.duration > 28800) {
errorHandler(new InvalidVideoDuration(url, vid.duration), level);
}
}
if (vid.rating !== undefined && (vid.rating < 0 || vid.rating > 5)) {
errorHandler(new InvalidVideoRating(url, vid.title, vid.rating), level);
}
if (typeof vid !== 'object' ||
!vid.thumbnail_loc ||
!vid.title ||
!vid.description) {
// has to be an object and include required categories https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190
errorHandler(new InvalidVideoFormat(url), level);
}
if (vid.title.length > 100) {
errorHandler(new InvalidVideoTitle(url, vid.title.length), level);
}
if (vid.description.length > 2048) {
errorHandler(new InvalidVideoDescription(url, vid.description.length), level);
}
if (vid.view_count !== undefined && vid.view_count < 0) {
errorHandler(new InvalidVideoViewCount(url, vid.view_count), level);
}
if (vid.tag.length > 32) {
errorHandler(new InvalidVideoTagCount(url, vid.tag.length), level);
}
if (vid.category !== undefined && vid.category?.length > 256) {
errorHandler(new InvalidVideoCategory(url, vid.category.length), level);
}
if (vid.family_friendly !== undefined &&
!isValidYesNo(vid.family_friendly)) {
errorHandler(new InvalidVideoFamilyFriendly(url, vid.family_friendly), level);
}
if (vid.restriction) {
if (!validators.restriction.test(vid.restriction)) {
errorHandler(new InvalidVideoRestriction(url, vid.restriction), level);
}
if (!vid['restriction:relationship'] ||
!isAllowDeny(vid['restriction:relationship'])) {
errorHandler(new InvalidVideoRestrictionRelationship(url, vid['restriction:relationship']), level);
}
}
// TODO price element should be unbounded
if ((vid.price === '' && vid['price:type'] === undefined) ||
(vid['price:type'] !== undefined && !isPriceType(vid['price:type']))) {
errorHandler(new InvalidVideoPriceType(url, vid['price:type'], vid.price), level);
}
if (vid['price:resolution'] !== undefined &&
!isResolution(vid['price:resolution'])) {
errorHandler(new InvalidVideoResolution(url, vid['price:resolution']), level);
}
if (vid['price:currency'] !== undefined &&
!validators['price:currency'].test(vid['price:currency'])) {
errorHandler(new InvalidVideoPriceCurrency(url, vid['price:currency']), level);
}
validate(vid, 'video', url, level);
});
}
return conf;
}
node_modules/sitemap/dist/esm/lib/xmllint.d.ts
Generated Vendored
+12
View File
@@ -0,0 +1,12 @@
import { Readable } from 'node:stream';
/**
* Verify the passed in xml is valid. Requires xmllib be installed
*
* Security: This function always pipes XML content via stdin to prevent
* command injection vulnerabilities. Never pass user-controlled strings
* as file path arguments to xmllint.
*
* @param xml what you want validated (string or Readable stream)
* @return {Promise<void>} resolves on valid rejects [error stderr]
*/
export declare function xmlLint(xml: string | Readable): Promise<void>;
node_modules/sitemap/dist/esm/lib/xmllint.js
Generated Vendored
+78
View File
@@ -0,0 +1,78 @@
import { existsSync } from 'node:fs';
import { resolve } from 'node:path';
import { execFile } from 'node:child_process';
import { XMLLintUnavailable } from './errors.js';
/**
* Finds the `schema` directory with robust path resolution.
* Searches from the project root directory using process.cwd().
* This works correctly regardless of whether the code is running from:
* - Source: lib/xmllint.ts
* - ESM build: dist/esm/lib/xmllint.js
* - CJS build: dist/cjs/lib/xmllint.js
* - Test environment
*
* @throws {Error} if the schema directory is not found
* @returns {string} the path to the schema directory
*/
function findSchemaDir() {
// Search for schema directory from project root
// This works in test, build, and source environments
const possiblePaths = [
resolve(process.cwd(), 'schema'), // From project root
resolve(process.cwd(), '..', 'schema'), // One level up
resolve(process.cwd(), '..', '..', 'schema'), // Two levels up
];
for (const schemaPath of possiblePaths) {
if (existsSync(schemaPath)) {
return schemaPath;
}
}
throw new Error(`Schema directory not found. Searched paths: ${possiblePaths.join(', ')}`);
}
/**
* Verify the passed in xml is valid. Requires xmllib be installed
*
* Security: This function always pipes XML content via stdin to prevent
* command injection vulnerabilities. Never pass user-controlled strings
* as file path arguments to xmllint.
*
* @param xml what you want validated (string or Readable stream)
* @return {Promise<void>} resolves on valid rejects [error stderr]
*/
export function xmlLint(xml) {
const args = [
'--schema',
resolve(findSchemaDir(), 'all.xsd'),
'--noout',
'-', // Always read from stdin for security
];
return new Promise((resolve, reject) => {
execFile('which', ['xmllint'], (error, stdout, stderr) => {
if (error) {
reject([new XMLLintUnavailable()]);
return;
}
const xmllint = execFile('xmllint', args, (error, stdout, stderr) => {
if (error) {
reject([error, stderr]);
}
resolve();
});
// Always pipe XML content via stdin for security
if (xmllint.stdin) {
if (typeof xml === 'string') {
// Convert string to stream and pipe to stdin
xmllint.stdin.write(xml);
xmllint.stdin.end();
}
else if (xml) {
// Pipe readable stream to stdin
xml.pipe(xmllint.stdin);
}
}
if (xmllint.stdout) {
xmllint.stdout.unpipe();
}
});
});
}
node_modules/sitemap/eslint.config.mjs
Generated Vendored
+102
View File
@@ -0,0 +1,102 @@
import { defineConfig, globalIgnores } from "eslint/config";
import jest from "eslint-plugin-jest";
import typescriptEslint from "@typescript-eslint/eslint-plugin";
import globals from "globals";
import tsParser from "@typescript-eslint/parser";
import path from "node:path";
import { fileURLToPath } from "node:url";
import js from "@eslint/js";
import { FlatCompat } from "@eslint/eslintrc";
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const compat = new FlatCompat({
baseDirectory: __dirname,
recommendedConfig: js.configs.recommended,
allConfig: js.configs.all
});
export default defineConfig([globalIgnores([
"test/",
"**/__test__",
"**/__tests__",
"**/node_modules",
"node_modules/",
"**/node_modules/",
"**/.idea",
"**/.nyc_output",
"**/coverage",
"**/*.d.ts",
"bin/**/*",
]), {
extends: compat.extends(
"eslint:recommended",
"plugin:@typescript-eslint/eslint-recommended",
"plugin:@typescript-eslint/recommended",
"prettier",
"plugin:prettier/recommended",
),
plugins: {
jest,
"@typescript-eslint": typescriptEslint,
},
languageOptions: {
globals: {
...globals.jest,
...globals.node,
},
parser: tsParser,
ecmaVersion: 2023,
sourceType: "module",
},
rules: {
indent: "off",
"lines-between-class-members": ["error", "always", {
exceptAfterSingleLine: true,
}],
"no-case-declarations": 0,
"no-console": 0,
"no-dupe-class-members": "off",
"no-unused-vars": 0,
"padding-line-between-statements": ["error", {
blankLine: "always",
prev: "multiline-expression",
next: "multiline-expression",
}],
"@typescript-eslint/ban-ts-comment": ["error", {
"ts-expect-error": "allow-with-description",
}],
"@typescript-eslint/explicit-member-accessibility": "off",
"@typescript-eslint/naming-convention": ["error", {
selector: "default",
format: null,
}, {
selector: "interface",
prefix: [],
format: null,
}],
"@typescript-eslint/no-parameter-properties": "off",
"@typescript-eslint/no-unused-vars": ["error", {
args: "none",
}],
},
}, {
files: ["**/*.js"],
rules: {
"@typescript-eslint/explicit-function-return-type": "off",
"@typescript-eslint/no-var-requires": "off",
},
}]);
node_modules/sitemap/jest.config.cjs
Generated Vendored
+36
View File
@@ -0,0 +1,36 @@
/** @type {import('jest').Config} */
const config = {
preset: 'ts-jest',
transform: {
'^.+\\.ts?$': [
'ts-jest',
{
tsconfig: 'tsconfig.jest.json',
diagnostics: {
ignoreCodes: [151002],
},
},
],
},
moduleNameMapper: {
'^(\\.{1,2}/.*)\\.js$': '$1',
},
modulePathIgnorePatterns: ['<rootDir>/dist/'],
collectCoverage: true,
collectCoverageFrom: [
'lib/**/*.ts',
'!lib/**/*.d.ts',
'!lib/xmllint.ts',
'!node_modules/',
],
coverageThreshold: {
global: {
branches: 80,
functions: 90,
lines: 90,
statements: 90,
},
},
};
module.exports = config;
node_modules/sitemap/package.json
Generated Vendored
+104
View File
@@ -0,0 +1,104 @@
{
"name": "sitemap",
"version": "9.0.1",
"description": "Sitemap-generating lib/cli",
"keywords": [
"sitemap",
"sitemap.xml"
],
"homepage": "https://github.com/ekalinin/sitemap.js#readme",
"bugs": {
"url": "https://github.com/ekalinin/sitemap.js/issues"
},
"repository": {
"type": "git",
"url": "git://github.com/ekalinin/sitemap.js.git"
},
"license": "MIT",
"author": "Eugene Kalinin <e.v.kalinin@gmail.com>",
"type": "module",
"exports": {
".": {
"import": {
"types": "./dist/esm/index.d.ts",
"default": "./dist/esm/index.js"
},
"require": {
"types": "./dist/cjs/index.d.ts",
"default": "./dist/cjs/index.js"
}
},
"./package.json": "./package.json"
},
"main": "./dist/cjs/index.js",
"module": "./dist/esm/index.js",
"types": "./dist/esm/index.d.ts",
"bin": "./dist/esm/cli.js",
"directories": {
"lib": "lib",
"test": "tests"
},
"scripts": {
"build": "npm run build:esm && npm run build:cjs-package && npm run build:cjs",
"build:cjs": "tsc -p tsconfig.cjs.json",
"build:cjs-package": "mkdir -p dist/cjs && echo '{\"type\":\"commonjs\"}' > dist/cjs/package.json",
"build:esm": "tsc",
"lint": "eslint \"{lib,tests}/**/*.ts\" ./cli.ts",
"lint:fix": "eslint --fix \"{lib,tests}/**/*.ts\" ./cli.ts",
"prepare": "husky",
"prepublishOnly": "rm -rf dist && npm run build && npm run test",
"prettier": "npx prettier --check \"{lib,tests}/**/*.ts\" ./cli.ts",
"prettier:fix": "npx prettier --write \"{lib,tests}/**/*.ts\" ./cli.ts",
"test": "jest",
"test:full": "npm run lint && npm run build && jest && npm run test:xmllint",
"test:perf": "node ./tests/perf.mjs",
"test:schema": "node tests/alltags.mjs | xmllint --schema schema/all.xsd --noout -",
"test:typecheck": "tsc",
"test:xmllint": "if which xmllint; then npm run test:schema; else echo 'skipping xml tests. xmllint not installed'; fi"
},
"lint-staged": {
"package.json": [
"sort-package-json"
],
"{lib,tests}/**/*.ts": [
"eslint --fix",
"prettier --write"
]
},
"dependencies": {
"@types/node": "^24.9.2",
"@types/sax": "^1.2.1",
"arg": "^5.0.0",
"sax": "^1.4.1"
},
"devDependencies": {
"@babel/core": "^7.28.5",
"@eslint/eslintrc": "^3.3.1",
"@eslint/js": "^9.39.0",
"@types/jest": "^30.0.0",
"@types/memorystream": "^0.3.0",
"@typescript-eslint/eslint-plugin": "^8.46.2",
"@typescript-eslint/parser": "^8.46.2",
"eslint": "^9.39.0",
"eslint-config-prettier": "^10.1.8",
"eslint-plugin-jest": "^29.0.1",
"eslint-plugin-prettier": "^5.1.3",
"express": "^5.1.0",
"globals": "^16.5.0",
"husky": "^9.0.11",
"jest": "^30.2.0",
"lint-staged": "^16.2.6",
"memorystream": "^0.3.1",
"prettier": "^3.2.5",
"sort-package-json": "^3.4.0",
"stats-lite": "^2.2.0",
"stream-json": "^1.7.1",
"through2-map": "^4.0.0",
"ts-jest": "^29.1.3",
"typescript": "^5.4.5"
},
"engines": {
"node": ">=20.19.5",
"npm": ">=10.8.2"
}
}
node_modules/sitemap/schema/all.xsd
Generated Vendored
+8
View File
@@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<schema elementFormDefault="qualified" xmlns="http://www.w3.org/2001/XMLSchema">
<import namespace="http://www.sitemaps.org/schemas/sitemap/0.9" schemaLocation="./sitemap.xsd" />
<import namespace="http://www.google.com/schemas/sitemap-video/1.1" schemaLocation="http://www.google.com/schemas/sitemap-video/1.1/sitemap-video.xsd"/>
<import namespace="http://www.google.com/schemas/sitemap-image/1.1" schemaLocation="http://www.google.com/schemas/sitemap-image/1.1/sitemap-image.xsd"/>
<import namespace="http://www.google.com/schemas/sitemap-news/0.9" schemaLocation="http://www.google.com/schemas/sitemap-news/0.9/sitemap-news.xsd"/>
<import namespace="http://www.w3.org/1999/xhtml" schemaLocation="http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd"/>
</schema>
node_modules/sitemap/schema/sitemap.xsd
Generated Vendored
+116
View File
@@ -0,0 +1,116 @@
<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
elementFormDefault="qualified">
<xsd:annotation>
<xsd:documentation>
XML Schema for Sitemap files.
Last Modifed 2008-03-26
</xsd:documentation>
</xsd:annotation>
<xsd:element name="urlset">
<xsd:annotation>
<xsd:documentation>
Container for a set of up to 50,000 document elements.
This is the root element of the XML file.
</xsd:documentation>
</xsd:annotation>
<xsd:complexType>
<xsd:sequence>
<xsd:any namespace="##other" minOccurs="0" maxOccurs="unbounded" processContents="strict"/>
<xsd:element name="url" type="tUrl" maxOccurs="unbounded"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<xsd:complexType name="tUrl">
<xsd:annotation>
<xsd:documentation>
Container for the data needed to describe a document to crawl.
</xsd:documentation>
</xsd:annotation>
<xsd:sequence>
<xsd:element name="loc" type="tLoc"/>
<xsd:element name="lastmod" type="tLastmod" minOccurs="0"/>
<xsd:element name="changefreq" type="tChangeFreq" minOccurs="0"/>
<xsd:element name="priority" type="tPriority" minOccurs="0"/>
<xsd:any namespace="##other" minOccurs="0" maxOccurs="unbounded" processContents="strict"/>
</xsd:sequence>
</xsd:complexType>
<xsd:simpleType name="tLoc">
<xsd:annotation>
<xsd:documentation>
REQUIRED: The location URI of a document.
The URI must conform to RFC 2396 (http://www.ietf.org/rfc/rfc2396.txt).
</xsd:documentation>
</xsd:annotation>
<xsd:restriction base="xsd:anyURI">
<xsd:minLength value="12"/>
<xsd:maxLength value="2048"/>
</xsd:restriction>
</xsd:simpleType>
<xsd:simpleType name="tLastmod">
<xsd:annotation>
<xsd:documentation>
OPTIONAL: The date the document was last modified. The date must conform
to the W3C DATETIME format (http://www.w3.org/TR/NOTE-datetime).
Example: 2005-05-10
Lastmod may also contain a timestamp.
Example: 2005-05-10T17:33:30+08:00
</xsd:documentation>
</xsd:annotation>
<xsd:union>
<xsd:simpleType>
<xsd:restriction base="xsd:date"/>
</xsd:simpleType>
<xsd:simpleType>
<xsd:restriction base="xsd:dateTime"/>
</xsd:simpleType>
</xsd:union>
</xsd:simpleType>
<xsd:simpleType name="tChangeFreq">
<xsd:annotation>
<xsd:documentation>
OPTIONAL: Indicates how frequently the content at a particular URL is
likely to change. The value "always" should be used to describe
documents that change each time they are accessed. The value "never"
should be used to describe archived URLs. Please note that web
crawlers may not necessarily crawl pages marked "always" more often.
Consider this element as a friendly suggestion and not a command.
</xsd:documentation>
</xsd:annotation>
<xsd:restriction base="xsd:string">
<xsd:enumeration value="always"/>
<xsd:enumeration value="hourly"/>
<xsd:enumeration value="daily"/>
<xsd:enumeration value="weekly"/>
<xsd:enumeration value="monthly"/>
<xsd:enumeration value="yearly"/>
<xsd:enumeration value="never"/>
</xsd:restriction>
</xsd:simpleType>
<xsd:simpleType name="tPriority">
<xsd:annotation>
<xsd:documentation>
OPTIONAL: The priority of a particular URL relative to other pages
on the same site. The value for this element is a number between
0.0 and 1.0 where 0.0 identifies the lowest priority page(s).
The default priority of a page is 0.5. Priority is used to select
between pages on your site. Setting a priority of 1.0 for all URLs
will not help you, as the relative priority of pages on your site
is what will be considered.
</xsd:documentation>
</xsd:annotation>
<xsd:restriction base="xsd:decimal">
<xsd:minInclusive value="0.0"/>
<xsd:maxInclusive value="1.0"/>
</xsd:restriction>
</xsd:simpleType>
</xsd:schema>
node_modules/sitemap/tsconfig.cjs.json
Generated Vendored
+9
View File
@@ -0,0 +1,9 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"module": "CommonJS",
"moduleResolution": "node10",
"outDir": "./dist/cjs/"
},
"exclude": ["node_modules", "cli.ts"]
}
node_modules/sitemap/tsconfig.jest.json
Generated Vendored
+9
View File
@@ -0,0 +1,9 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"resolveJsonModule": true,
"lib": ["es2018", "dom"],
"forceConsistentCasingInFileNames": true,
"noEmit": true
}
}