Move pdpv0 into main branch #840
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* feat(pdp): commpv2 shim Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix(pdp): commpv2 shim more * fix(pdp): commpv2 fix upload commp digest comparison * feat(pdp): update pdptool to do commpv2 * fix(pdp): update contract address --------- Signed-off-by: Jakub Sztandera <oss@kubuxu.com> Co-authored-by: Jakub Sztandera <oss@kubuxu.com>
* fix(pdp): use pgx.ErrNoRows * chore(pdp): update contract address
* generate .abi from .json * update abi * abigen with make * extract package name from stem * rename * use GetPDPConfig * fix capitalization
…614) Treat it as indirection to the contract that we can use to find a contract that implements IPDPProvingSchedule that we can use for proving parameters.
Using current Alpha release FilecoinWarmStorageService contract address on calibnet Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com>
* feat: ui node cordon button * improve visuals
* move proofsvc here * layout tasks * request task * missing sql schema * implement prove * proofshare: Submit task * proofshare webrpc * proofshare web * make gen * fixes for basic flow * client schema/ui * improve market ui more * pure Go post proof verification * proof bincode encode * bg task name * wip genadt * cborlink * update cbor-gen * init router * debug commands * working deposit * most flows working * Withdraw, DST * Complete Withdraw * more info * Plumb most of the remaining stuff * router wallet ui start * webui balance add * make gen * client message listing * usable withdrawal * ui: price settings * pricing integration * sort of working c1outraw cbor * undo * wip pricing changes * voucher check utils * persist provider payments * wip * wip remote task pipeline * cleanup client * guided setup for non-sp cluster * correct enable * make gen * wallet source fixes * don't retry bad batch tasks for no reason * missing clientctl * closer to working * payment recovery * fix payment undo * ocean of fixes * make gen * basic backoffs * record provider payments correctly * some settle ui * redeem improvements * ui: wallet hover details * provider settle button * minor fixes * more submit retry * snap vanilla verif on proof type * missing files * snap snark check * remote snap prove task * snap pricing * close to final provider ui * gen, new ledger types * more types * block helpers * svc l2 iteration * make gen * svc contract final touches * snksvc client cli * snk ask withdraw * dedup reassigned requests * handle pow * prove oldest first * sig msg type, gen * signing impl * basic noncecache * signer fixes * wire up signatures * make gen * tos * tos in schema * deployment settings * Apply suggestions from code review Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com> * tos finalization * add licence term to psvc packages * make gen * rm temp task skip * fix initial provider form * remove useless price epoch from proof request * gone work handling * cache client read calls * fixed client price math * log zero price * pricing debug * more pricinc debug * fix price cache * caches are hard * allow separate cql db * register proving tasks when remote proving is enabled * what * more detailed logs * filter market prices * good logs are good * Provider autosettle * fix infer nvme devices * Don't auto-accept AI suggestions * Wallet Balance Manager * gofmt * make gen * web: compact pretty epoch in porep pipeline * del unused scripts/gen.go * lint fixes * faster poll * poll metrics * client payments table stats * provider: smarter sched logic * hold the correct way * attempt at smart storage allocation * storage tuning * rm bs fetch limit * fix likely deadlock * refactor client pipeline * gen, default upload max * review * fix orphan send * fix name len limit * fix null in psvc poll * eww * fix porep get * fix snap too * fix poll edge-cases * send fixes * filter spdis in createpayment correctly * fix bad src address * fix proof retry hopefully * optimized snark candidate picker * mark finished submit when orphaned * make gen * cleanup abandoned tasks * more generous retry * fix lint * drop default hardhat readme * validate wallet input strings * fix ctxlk test * fix chainhead ddos * address review * make gen * translations * gen translations * cleanup stray todo --------- Co-authored-by: Andrew Jackson (Ajax) <snadrus@gmail.com>
* basic wallet exporter * plumb into the constructor * fix lint * increasing bucket order * make gen * move wallet exporter config to subsystems * missing file * docs * post-merge make gen
* per sector commit simulation * fix batching timezone
* sector sched UI * confirm btn * integrate into supraseal * fix table * make gen * don't start batch seal with too-few sectors * rename scheduler upsert rpc to edit
* snark market balancemgr ui/schema * balancemgr filter supported types in CanAccept * implement proofshare bmgr do * make gen * cleanup * fix msg tracking address type
* cordon: yield bg tasks * make PSClientPoll yield as well * harmonytask: Scheduling Overrides * metrics: record version * fix build * smart restart * webui: Jsonrpc reconnect fixes * webui: restart requests * make gen * jrpc reject on conn fail * missing schema file * rm snake case
* release v1.26.0 * fix FFI version
Specifying @filecoin-project/curio as the codeowners so that branch protection rules / rulesets against `main` that "Require review from Code Owners" actually has enforcement.
) * fix(pdp): foreign keys pointing at pdp_piecerefs cascade on delete Both of these tables user `not null` for that foreign key so it means that deleting rows from pdp_piecerefs was impossible Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * chore: metnion changes of schema in old schma files Signed-off-by: Jakub Sztandera <oss@kubuxu.com> --------- Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Add remote addr, query string and headers.
…759) Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* chore(pdp): tag task failures with dataset id Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * chore: go-fmt Signed-off-by: Jakub Sztandera <oss@kubuxu.com> --------- Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* fix(pdp): encoding of zero integer capabilities For one reason or another we require zeros to be encoded as 1 byte slices. Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * chore: rename encodeBigInt -> encodeBigIntCapability Signed-off-by: Jakub Sztandera <oss@kubuxu.com> --------- Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* fix(pdp): don't try proving empty datasets Empty datasets set proving epoch to zero (NO_CHALLENGE_SCHEDULED). Number of leaves is also zero, which was causing #761 . This creates a bit of an edge case when piece deletions, additions and proving happen in the same time window: - data set gets used, pieces are added and proven - all pieces are deleted from it - nextProvingPeriod gets called on an empty dataset - a new piece gets added, it sets `init_ready = TRUE`, but it is true already - prove task fires, detects that proving set is empty, challenge epoch is 0 (as the proving set is empty), proving gets disabled Now the dataset won't get proven until one more piece gets added to set `init_ready = TRUE`. Better pattern here would be to react to events emitted in our messages from the transactions we send to PDPVerifier. As ordering can get even more tricky if you consider that transactions are sent async. Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * chore: gofmt I had fmt_on_save accidentally disabled Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * chore(pdp): include dataSetId in NextPP task log Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix: disable proving query NUL -> NULL Signed-off-by: Jakub Sztandera <oss@kubuxu.com> --------- Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
…#772) Who knew that if you say that nothing should be indexed, nothing will get indexed. Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Decode extraData in handleCreateDataSetAndAddPieces to detect withIPFSIndexing metadata and mark pieces for indexing, matching behavior of separate addPieces endpoint.
* fix(pdp): delay proving by one epoch In logs sometimes we observed that the proving was happening to early, possible cause is reorgs knocking off a head, causing the message to land a block early. Resolution: delay proving by one epoch The task will be re-tried in any case but this will reduce noise. Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * chore(pdp): explicity log proving related failures While these failures get logged through taskhandler, log them explicitly to get structured logs with `dataSetId`. Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix(pdp): if we encounter ProvingPeriodNotInitialized, move back to init Sometimes for a not fully understood reasons (my guess is deletion), proving period can get de-initialized. This causes NextProvingPeriodTask to loop in error state over and over. Resolve it by moving from proving loop to init state. Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix(pdp): in proving period init state, don't re-try if dataset is empty We used to trigger PPInit early, before pieces landed. We shifted to doing it only after pieces land. If we encounter dataset with no pieces in init state, it should be safe to assume that the leaf count is accurate. Signed-off-by: Jakub Sztandera <oss@kubuxu.com> --------- Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* settle lockup and dataSet delete * minor fixes * fix imports * fix payee * apply suggestions from review * apply review suggestions * fix SQL * remove unused param * update settle task frequency * feat: piece deletion and cleanup (#771) * piece deletion and cleanup * use join * fix IPNI cleanup * add piece_cid index to pdp_piecerefs table * fix args * more conditional checks * move delete lower * fix deletion logic * delete conditionally
* chore: remove trailing spaces Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix(pdp): pdss -> pdds in processPendingDeletes Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix(pdp): watch_piece_delete rm_message_hex -> rm_message_hash Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix(pdp): processIndexingAndIPNICleanup, piece_size -> piece_padded_size Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix(pdp): processIndexingAndIPNICleanup, piece_size -> piece_padded_size Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix(pdp): metadata not existing, it is just default IPFS gateway meta Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix(pdp): the query uses ID not PieceRef Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * chore: fix imports Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix(pdp): missing continue if msg status is not known Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * fix(pdp): update rm_message_hash in handleDeleteDataSetPiece Signed-off-by: Jakub Sztandera <oss@kubuxu.com> --------- Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* fix(pdp): commit removal transaction when we don't have IPNI ads Currently the transaction will be rolled back when there are no IPNI ads for the pieces. Signed-off-by: Jakub Sztandera <oss@kubuxu.com> * chore(pdp): make cleaning up log more accurate Signed-off-by: Jakub Sztandera <oss@kubuxu.com> --------- Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Just misc error and log output fixes Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
Signed-off-by: Jakub Sztandera <oss@kubuxu.com>
* fix: align address decoding with Solidity canonical behavior * fix(pdp): Align capability address decoding to match Solidity behavior (Fixes #758) * Minor changes * Changes * Logic change
* return 404 for piece retrievals correctly * fix multiple wrapped errors with xerrors.Errorf Signed-off-by: Jakub Sztandera <oss@kubuxu.com> --------- Signed-off-by: Jakub Sztandera <oss@kubuxu.com> Co-authored-by: Jakub Sztandera <oss@kubuxu.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Preserve the original PieceCID from the URL and use it for the Content-Disposition header filename. Fixes #824 Signed-off-by: Anubhav Singh <anubhavsingh@Anubhavs-MacBook-Air.local> Co-authored-by: Anubhav Singh <anubhavsingh@Anubhavs-MacBook-Air.local>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.