Anthrove/plug-sdk
5
0
Fork 0
Code Issues Pull Requests Actions Projects Releases 10 Activity

New and generic scrape algorithm #11

Merged
SoXX merged 20 commits from dev/algorithm into main 2024-11-01 21:39:21 +00:00
Conversation 0 Commits 20 Files Changed 8 +244 -87
SoXX commented 2024-10-26 19:15:51 +00:00
Owner
Copy Link
No description provided.
SoXX added 5 commits 2024-10-26 19:15:52 +00:00
feat: Refactor task execution to use Plug interface and update algorithm implementation
All checks were successful
Gitea Build Check / Build (push) Successful in 1m3s
Details
41882b9bfb
fix: Capitalize struct fields for public access
All checks were successful
Gitea Build Check / Build (push) Successful in 1m2s
Details
107a317095
fix: nil pointer for DB
All checks were successful
Gitea Build Check / Build (push) Successful in 57s
Details
408c977432
fix: Remove redundant error logging and tracing in TaskStart function
All checks were successful
Gitea Build Check / Build (push) Successful in 56s
Details
36d6ecd873
feat: implemented generic algorithm
All checks were successful
Gitea Build Check / Build (push) Successful in 57s
Details
Gitea Build Check / Build (pull_request) Successful in 55s
Details
376d7ba75d
SoXX requested review from Alphyron 2024-10-26 19:15:55 +00:00
Alphyron requested changes 2024-10-26 19:20:16 +00:00
Dismissed
pkg/plug/algorithm.go Outdated
@ -0,0 +61,4 @@
nextPage := ""
outer:
for anthroveUserFavCount < profile.UserFavoriteCount {
Alphyron commented 2024-10-26 19:18:16 +00:00
Owner
Copy Link

Hier muss eine sonderbedingung gemacht weerden, weil UserFavoriteCount kann ja -1 sein (wenn es nicht ausgelesen werden kann.

Hier muss eine sonderbedingung gemacht weerden, weil UserFavoriteCount kann ja -1 sein (wenn es nicht ausgelesen werden kann.
Alphyron commented 2024-10-26 20:11:41 +00:00
Owner
Copy Link

@SoXX to fix this problem and the Issue one comment bellow I would recommend, to change the algorithm to the following:

@SoXX to fix this problem and the Issue one comment bellow I would recommend, to change the algorithm to the following:
Alphyron commented 2024-10-26 20:46:41 +00:00
Owner
Copy Link

https://www.plantuml.com/plantuml/uml/RP1FRzH03CNl_XIlj-mDhea4BIgWIXMAgChTk-D9HyoVoEFija9yTtOIYTBIkOtzUy_Flgp6QakAT34hJucnLFaXQk70ySQZPA8LeVwsiCDz5Hsr-105Nal269VfQhmPwFJGQftf8ZiwFw3_AeOlV1nvsk1LFH2mUSbZg1RoXB5KgnlHs7__rv_-vvkdRFqtFPgcYQwSGoxsch62APOzHypdF-AvERo9RsEUSS_7bKPNr8aYL8Gq5pNETe4i9wa67xJQRa1B43owpo_TFk3T3hy80FOg_9E0tslOQ_4X2xx9esr7i8AxW_8i0qbswtM9-liv5cuvywkr1kg_or6qIfk4spLdBYUKw9vpzNyTjZXTi1Thm1v4fPKODN6CSC5xbNmGxCLE8haX2LtYfxtWFLBzkDji7Pj0nHRDk5jIOdtgYQgLcIubkoN5Fm00

[https://www.plantuml.com/plantuml/uml/RP1FRzH03CNl_XIlj-mDhea4BIgWIXMAgChTk-D9HyoVoEFija9yTtOIYTBIkOtzUy_Flgp6QakAT34hJucnLFaXQk70ySQZPA8LeVwsiCDz5Hsr-105Nal269VfQhmPwFJGQftf8ZiwFw3_AeOlV1nvsk1LFH2mUSbZg1RoXB5KgnlHs7__rv_-vvkdRFqtFPgcYQwSGoxsch62APOzHypdF-AvERo9RsEUSS_7bKPNr8aYL8Gq5pNETe4i9wa67xJQRa1B43owpo_TFk3T3hy80FOg_9E0tslOQ_4X2xx9esr7i8AxW_8i0qbswtM9-liv5cuvywkr1kg_or6qIfk4spLdBYUKw9vpzNyTjZXTi1Thm1v4fPKODN6CSC5xbNmGxCLE8haX2LtYfxtWFLBzkDji7Pj0nHRDk5jIOdtgYQgLcIubkoN5Fm00](https://plantuml.github.io/plantuml-core/raw.html?VLB1RXCn4BtxArulrUu5qMiMWGejg5A5LbJbxcJ76mlRinhF9ktVOsS3j57epPxtvisRPrzCOiTeCMVwIQ9-OyET0oTZibfHixfdE-0na_J2pWU24uxempnUztUTsMzhhN4ij2DfZG6yGsEJReLtL1k_sjqNdfFY0wP5uzsnpEAL5kpoyWtm8zwtq2qbD2epjMK8i1Qolyg9qk1TdRlwHehIsDlnwB8gTHKvB45FFgjF8thEcezxLEwB-ytdC_oYIKVyd4RjgKtNmu34UifHnXXuRGEcSaSeO7UMlepvHtTsTs2ZOHb8u3Dyd5YqX7k1H7igcFEIjUtm9_ZRrsgdoU4_qTsGyudmdPvqoQJOUDY8dQWpXNLuPMDMZrEeoIHE9rjsPlMZVe1CTJ3k3xOJxy5XU67phyBYhl7wQglgLAYMB7Aq8Q3uSq9fGrYJxgdiKIWHJRJFQy8LOlxKBhA3LOAGpfBmCCTevRchdvYdtm00)
Alphyron commented 2024-10-26 20:55:47 +00:00
Owner
Copy Link

fixed an issue @SoXX

https://plantuml.github.io/plantuml-core/raw.html?ZL9DJzmm4BtxLpmkABbKzBefL53Q2WbLKH6zUpVZh5N74uqdk_3lQoTBAGkjUeldVSoRoPoCOll1OahqWqJzneOR1ux69BMYPdNBjiDz8cc5dGy49poW3LD_sTuqPhyjjSgnq8waDWRm3fMDkXNUKH5-iRjFF4N51uoBnxj3cSKhBTZfwJ_02vpLq2r5L2eJrRa9i1QoF_CNnSwxqdVrbHHPsDh-aB8uDK-HvdBBIHFEpuyCyHV7UNXwy4bzx0_YPVvdi_bzVTMyvkSiTC3VyYYQ8hhiEaJMOuuo-i1h6p3cDq86cpVfIvjdkhExcOsAfIE1J_33oABvfvmWqfuovhmahHiyRUVpIxUB_rpGDJaV2T_eGPUTv1Xt6x4ZDIQpvkimPdK_XhAPoF5eoiQilfV3ILGQutQE7NaF3jqVCNyxBEagho-cAac5IiaY3JO4v7nEA6u8Qz8zvJaoPK0rsCSruJs6zB5UR3kh126T9E9k6WOhZqqvVXv_0G00

fixed an issue @SoXX https://plantuml.github.io/plantuml-core/raw.html?ZL9DJzmm4BtxLpmkABbKzBefL53Q2WbLKH6zUpVZh5N74uqdk_3lQoTBAGkjUeldVSoRoPoCOll1OahqWqJzneOR1ux69BMYPdNBjiDz8cc5dGy49poW3LD_sTuqPhyjjSgnq8waDWRm3fMDkXNUKH5-iRjFF4N51uoBnxj3cSKhBTZfwJ_02vpLq2r5L2eJrRa9i1QoF_CNnSwxqdVrbHHPsDh-aB8uDK-HvdBBIHFEpuyCyHV7UNXwy4bzx0_YPVvdi_bzVTMyvkSiTC3VyYYQ8hhiEaJMOuuo-i1h6p3cDq86cpVfIvjdkhExcOsAfIE1J_33oABvfvmWqfuovhmahHiyRUVpIxUB_rpGDJaV2T_eGPUTv1Xt6x4ZDIQpvkimPdK_XhAPoF5eoiQilfV3ILGQutQE7NaF3jqVCNyxBEagho-cAac5IiaY3JO4v7nEA6u8Qz8zvJaoPK0rsCSruJs6zB5UR3kh126T9E9k6WOhZqqvVXv_0G00
Alphyron marked this conversation as resolved
pkg/plug/algorithm.go Outdated
@ -0,0 +62,4 @@
outer:
for anthroveUserFavCount < profile.UserFavoriteCount {
select {
Alphyron commented 2024-10-26 19:18:39 +00:00
Owner
Copy Link

Issue still exists
Anthrove/plug-e621#15

Issue still exists https://git.anthrove.art/Anthrove/plug-e621/issues/15
Alphyron marked this conversation as resolved
pkg/plug/server.go Outdated
@ -100,3 +99,2 @@
func SetTaskExecutionFunction(function TaskExecution) {
taskExecutionFunction = function
func SetTaskExecutionFunction(plugInterface Plug) {
Alphyron commented 2024-10-26 19:20:13 +00:00
Owner
Copy Link

irgendwie passt der Name der funktion nicht zu dem was gemacht wird. Es wird weder eine Function gesetzt, noch wird ein execution stuff gesetzt.
Hier ist es ja nur ein Interface als Schnittstelle zu der PlugAPI

irgendwie passt der Name der funktion nicht zu dem was gemacht wird. Es wird weder eine Function gesetzt, noch wird ein execution stuff gesetzt. Hier ist es ja nur ein Interface als Schnittstelle zu der PlugAPI
Alphyron marked this conversation as resolved
SoXX added 1 commit 2024-10-26 19:31:42 +00:00
refactor: Rename SetTaskExecutionFunction to RegisterPlugInterface for clarity and add docstring
All checks were successful
Gitea Build Check / Build (push) Successful in 53s
Details
Gitea Build Check / Build (pull_request) Successful in 54s
Details
1435ae8ea4
SoXX added 1 commit 2024-10-26 21:13:31 +00:00
feat: start implementing new algorithm
All checks were successful
Gitea Build Check / Build (push) Successful in 1m2s
Details
Gitea Build Check / Build (pull_request) Successful in 1m4s
Details
b2db0664d6
SoXX added 1 commit 2024-10-28 20:09:26 +00:00
feat: Add tracing events and logging for batch processing steps
All checks were successful
Gitea Build Check / Build (push) Successful in 58s
Details
Gitea Build Check / Build (pull_request) Successful in 57s
Details
44137a7251
SoXX added 1 commit 2024-10-28 20:29:35 +00:00
fix: Fix conditional check for anthroveFaves length
All checks were successful
Gitea Build Check / Build (push) Successful in 59s
Details
Gitea Build Check / Build (pull_request) Successful in 56s
Details
263d65adc1
SoXX added 1 commit 2024-10-28 20:33:53 +00:00
fix: variable scope and append results in BatchPostProcessing
All checks were successful
Gitea Build Check / Build (push) Successful in 54s
Details
Gitea Build Check / Build (pull_request) Successful in 56s
Details
16846c59c4
SoXX added 1 commit 2024-11-01 20:06:25 +00:00
refactor: improve readability and efficiency
Some checks failed
Gitea Build Check / Build (push) Failing after 31s
Details
Gitea Build Check / Build (pull_request) Failing after 27s
Details
799f943650 
- Remove nested for loop and redundant checks
- Streamline error handling and logging
- Rename getAnthrovePost to getAnthrovePostReferences for clarity
- Simplify BatchPostProcessingWithSummery to return only necessary values
SoXX added 1 commit 2024-11-01 20:10:29 +00:00
fix: Remove unused BatchSummery return values in error cases
All checks were successful
Gitea Build Check / Build (push) Successful in 44s
Details
Gitea Build Check / Build (pull_request) Successful in 41s
Details
996ec30de1
SoXX added 1 commit 2024-11-01 20:15:54 +00:00
fix: comparison logic in algorithm to correctly break the loop
All checks were successful
Gitea Build Check / Build (push) Successful in 41s
Details
Gitea Build Check / Build (pull_request) Successful in 41s
Details
9d950cf31d
SoXX added 1 commit 2024-11-01 20:29:42 +00:00
fix: Remove redundant break statement in taskSummery block
All checks were successful
Gitea Build Check / Build (push) Successful in 40s
Details
Gitea Build Check / Build (pull_request) Successful in 40s
Details
0c0d8451c6
SoXX added 2 commits 2024-11-01 20:43:25 +00:00
Merge branch 'dev/algorithm' of git.anthrove.art:Anthrove/plug-sdk into dev/algorithm
All checks were successful
Gitea Build Check / Build (push) Successful in 38s
Details
Gitea Build Check / Build (pull_request) Successful in 38s
Details
0224a43736
SoXX added 1 commit 2024-11-01 21:14:41 +00:00
fix: favorites creation timestamps and remove duplicate imports
Some checks failed
Gitea Build Check / Build (push) Failing after 25s
Details
Gitea Build Check / Build (pull_request) Failing after 24s
Details
4d209f2a90
SoXX added 1 commit 2024-11-01 21:15:52 +00:00
fix: unused variable warning in post-processing loop
All checks were successful
Gitea Build Check / Build (push) Successful in 39s
Details
Gitea Build Check / Build (pull_request) Successful in 38s
Details
282e769c13
SoXX added 1 commit 2024-11-01 21:29:28 +00:00
fix: error when userID is empty
All checks were successful
Gitea Build Check / Build (push) Successful in 37s
Details
Gitea Build Check / Build (pull_request) Successful in 36s
Details
ffd3767f32 
- a error could happen if the user dose not have a source UserID/UserName set
SoXX added 1 commit 2024-11-01 21:34:53 +00:00
fix: variable userSource.UserID to userSource.AccountID for clarity
All checks were successful
Gitea Build Check / Build (push) Successful in 39s
Details
Gitea Build Check / Build (pull_request) Successful in 38s
Details
af0dc2ab0d
Alphyron approved these changes 2024-11-01 21:39:14 +00:00
SoXX merged commit af0dc2ab0d into main 2024-11-01 21:39:21 +00:00
SoXX deleted branch dev/algorithm 2024-11-01 21:39:21 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Alphyron
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Anthrove/plug-sdk#11
No description provided.
Delete Branch "dev/algorithm"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?